Le_Forgeron <jgr### [at] freefr> wrote:
> Btw, there is, in current main branch, an array sized by
> MAX_STRING_LEN_FAST, so this increase to 65536 is not a good idea at
> all. And limiting fast string to 256 is really what is intended (fast
> string is used for token, you do not want to type a 10235 charaters long
> identifier anyway). It's just the slower string that must be unbounded.
> (as long as your C++ memory allows it)

If MAX_STRING_LEN_FAST is used for an array, then it's indeed not a
very good idea to set it to 65536 because it would mean that even
single-character strings would start consuming that much RAM.

Using std::string for all strings is probably a very viable option.
Most std::string implementations nowadays use short string optimization
(although there are probably a few exceptions), so they are very
efficient for short strings, and probably what MAX_STRING_LEN_FAST
is already doing.

-- 
                                                          - Warp

Post a reply to this message

Warp <war### [at] tagpovrayorg> wrote:
> Le_Forgeron <jgr### [at] freefr> wrote:
> > Btw, there is, in current main branch, an array sized by
> > MAX_STRING_LEN_FAST, so this increase to 65536 is not a good idea at
> > all. And limiting fast string to 256 is really what is intended (fast
> > string is used for token, you do not want to type a 10235 charaters long
> > identifier anyway). It's just the slower string that must be unbounded.
> > (as long as your C++ memory allows it)
>
> If MAX_STRING_LEN_FAST is used for an array, then it's indeed not a
> very good idea to set it to 65536 because it would mean that even
> single-character strings would start consuming that much RAM.

> Using std::string for all strings is probably a very viable option.

The place where it is used is the core part of the tokenizer. Using a
dynamically allocated string would significantly increase overhead in the most
called place as the short string optimization won't work (it is frequently not
more than 10/14 chars). Now consider the average POV token length, and you
notice
you get at least one additional memory allocation per token parsed.

The only limit is that this can't parse character strings over 256 chars long.
However, as you can concat strings to any length in POV, this is neither a bug
now something that needs to be fixed. To the contrary...

Post a reply to this message

Thorsten Froehlich <nomail@nomail> wrote:
> The place where it is used is the core part of the tokenizer. Using a
> dynamically allocated string would significantly increase overhead in the most
> called place as the short string optimization won't work (it is frequently not
> more than 10/14 chars). Now consider the average POV token length, and you
> notice
> you get at least one additional memory allocation per token parsed.

Does the part that parses tokens really need to use the exact same string
class as the part that parses and handles strings?

> However, as you can concat strings to any length in POV, this is neither a bug
> now something that needs to be fixed. To the contrary...

Using concat to get longer string literals is a kludge.

-- 
                                                          - Warp

Post a reply to this message

Warp <war### [at] tagpovrayorg> wrote:
> Thorsten Froehlich <nomail@nomail> wrote:
> > The place where it is used is the core part of the tokenizer. Using a
> > dynamically allocated string would significantly increase overhead in the most
> > called place as the short string optimization won't work (it is frequently not
> > more than 10/14 chars). Now consider the average POV token length, and you
> > notice
> > you get at least one additional memory allocation per token parsed.
>
> Does the part that parses tokens really need to use the exact same string
> class as the part that parses and handles strings?

No, and actually, checking the code, this part of the tokenizer uses the unbound
string parsing. Which leaves the question how the original poster could get this
error message. The problem most likely is an identifier longer than 255
characters ... which is, well, really pointless to have.

Post a reply to this message

Le 30/12/2014 01:01, Thorsten Froehlich a écrit :
> Warp <war### [at] tagpovrayorg> wrote:
>> Thorsten Froehlich <nomail@nomail> wrote:
>>> The place where it is used is the core part of the tokenizer. Using a
>>> dynamically allocated string would significantly increase overhead in the most
>>> called place as the short string optimization won't work (it is frequently not
>>> more than 10/14 chars). Now consider the average POV token length, and you
>>> notice
>>> you get at least one additional memory allocation per token parsed.
>>
>> Does the part that parses tokens really need to use the exact same string
>> class as the part that parses and handles strings?
>
> No, and actually, checking the code, this part of the tokenizer uses the unbound
> string parsing. Which leaves the question how the original poster could get this
> error message. The problem most likely is an identifier longer than 255
> characters ... which is, well, really pointless to have.
>
>
Sorry dear, but a string literal (e.g.  "Povray is a nice program" in a 
#declare or #local) longer than 255 would lead to a parse error with the 
original parser.

Hence my patch in my master branch of hgpovray (made in May 2014).

I mean:

#declare MyString = "Povray is a nice program";

Now extends the string... and see povray fails to parse it when the 
limit is reached.
-- 
Just because nobody complains does not mean all parachutes are perfect.

Post a reply to this message

Le_Forgeron <lef### [at] freefr> wrote:
> Sorry dear, but a string literal (e.g.  "Povray is a nice program" in a
> #declare or #local) longer than 255 would lead to a parse error with the
> original parser.
>
> Hence my patch in my master branch of hgpovray (made in May 2014).
>
> I mean:
>
> #declare MyString = "Povray is a nice program";
>
> Now extends the string... and see povray fails to parse it when the
> limit is reached.

Hmm, then I probably fixed this in my local branch in Perforce some time ago
when I changed the code there to use UTF8 throughout ... the fix certainly is to
change the parsing code for string literals and not tweak the buffer size.

Post a reply to this message

Le 30/12/2014 16:54, Thorsten Froehlich a écrit :
> Le_Forgeron <lef### [at] freefr> wrote:
>> Sorry dear, but a string literal (e.g.  "Povray is a nice program" in a
>> #declare or #local) longer than 255 would lead to a parse error with the
>> original parser.
>>
>> Hence my patch in my master branch of hgpovray (made in May 2014).
>>
>> I mean:
>>
>> #declare MyString = "Povray is a nice program";
>>
>> Now extends the string... and see povray fails to parse it when the
>> limit is reached.
> 
> Hmm, then I probably fixed this in my local branch in Perforce some time ago
> when I changed the code there to use UTF8 throughout ... the fix certainly is to
> change the parsing code for string literals and not tweak the buffer size.
> 
> 
So does my own fix. I agree that the buffer size should not be tweaked.

Post a reply to this message

in news:54a2ebe3$1@news.povray.org Le_Forgeron wrote:

>> Hmm, then I probably fixed this in my local branch in Perforce some
>> time ago [...]
>> 
>> 
> So does my own fix. I agree that the buffer size should not be
> tweaked. 
> 

Just ran into this. (3.8.0-alpha.9945627+av625.msvc14.win64) 

As there seem to be various fixes available, will it be fixed in 3.8?

ingo

Post a reply to this message

ingo <ing### [at] tagpovrayorg> wrote:
> in news:54a2ebe3$1@news.povray.org Le_Forgeron wrote:
> ...
> Just ran into this. (3.8.0-alpha.9945627+av625.msvc14.win64)

the code below fails with all versions after 3.6.1 tried.

> As there seem to be various fixes available, will it be fixed in 3.8?

I just checked, this thread was started 4 years ago, to the day.  :-(



#version 3.6;

global_settings {
  assumed_gamma 1.0
  charset utf8
}

#declare S = "0123456789abcdef";
#declare M = "";

#declare I = 0;
#while(16 > I)
  #declare M = concat(M,S);
  // comment this + no output at all.
//  #debug concat("{",str(strlen(M),0,0), "} :", M, "\n")
  #declare I = I + 1;
#end

#debug concat("{",str(strlen(M),0,0), "} :", M)


(from strlen(S) = 208 the display goes wrong)


regards, jr.

Post a reply to this message

"jr" <cre### [at] gmailcom> wrote:

> (from strlen(S) = 208 the display goes wrong)
              ^^^
               M
>
> regards, jr.

Post a reply to this message