On 8/8/21 6:54 PM, clipka wrote:
> We have established that the `datetime(now)` combo is both currently 
> broken (on some systems), and has some shortcomings in its design (on 
> all systems, both broken and otherwise).
> 
> 
> I think the key issues in finding any satisfactory solution are the 
> following:
...

Suppose I'd lean toward (C.2) with (A.1) for what you've proposed.

FWIW.
----
In June I worked some on datetime() to support an internal always local 
clock now, but as importantly to make work the %z and %Z formatting 
options whether the fractional days were coming from now or being 
generated on the fly. The last bit, I failed initially to recall on 
mentioning the trailing 'Z' with a non utc time result.

Beyond the initial changes, I had plans for a new f_clock() internal 
function. Admittedly, my aim was in part to insulate povr from whatever 
'now' was doing, but there were additional reasons. Due your post, I've 
worked up an initial version of f_clock() much earlier than planned to 
test how crazy my plans were and to help clarify my thoughts.

---
In your preamble, you didn't mention 'now' being used as an SDL timing 
mechanism. It's something people do today and it carries risk with 
respect to any particular timing result. The reasons is we are using the 
system_clock for 'now' which can be updated at any time moving forward 
or backward - synchronization with time servers, etc.

We don't warn against timing use of 'now' in the documentation as far as 
I know. Perhaps we should?

In povr's f_clock() the up front returned time comes from steady_clock 
instead - which on my Ubuntu 20.04 machine appears to be a monotonic 
clock since the last reboot/boot.

Suppose the following povr SDL:

//---
#declare Fn_TimePt00 = function (sec_in_day) { now*sec_in_day }
#declare TimePt00 = f_clock(0,0,0);
#debug concat("f_clock(0,0,0) ",str(f_clock(0,0,0),9,9),"\n")
#for (I,0,3000000,1)
     #declare Dummy = 1.0;
#end
#declare Fn_TimePt01 = function (sec_in_day) { now*sec_in_day }
#declare TimePt01 = f_clock(0,0,0);
#debug concat("f_clock(0,0,0) ",str(f_clock(0,0,0),9,9),"\n")

#debug "\n"
#declare SecInDay = (24*60*60);
#debug concat("Fn_TimePt00 ",str(Fn_TimePt00(SecInDay),9,9),"\n")
#debug concat("Fn_TimePt01 ",str(Fn_TimePt01(SecInDay),9,9),"\n")
#debug concat("Fn_TimePt01 - Fn_TimePt00 ",
     str(Fn_TimePt01(SecInDay)-Fn_TimePt00(SecInDay),9,9),"\n")

#debug "\n"

#debug concat("TimePt00 ",str(TimePt00,9,9),"\n")
#debug concat("TimePt01 ",str(TimePt01,9,9),"\n")
#debug concat("TimePt01 - TimePt00 ",
     str(TimePt01-TimePt00,9,9),"\n")
#debug "\n"

#error "Parsing test. Stop early."
//---

The povr branch generates:

f_clock(0,0,0) 234835.462146713
Parsing 17764K tokens
f_clock(0,0,0) 234838.496408219

Fn_TimePt00 681978977.077232242
Fn_TimePt01 681978980.111495137
Fn_TimePt01 - Fn_TimePt00 3.034262896

TimePt00 234835.462139445
TimePt01 234838.496401535
TimePt01 - TimePt00 3.034262090

Of note for POV-Ray v3.8 and onward is that the Fn_TimePt00, Fn_TimePt00 
method for capturing time points should work in v3.8 beta 1, but it 
doesn't because when 'now' was added, the support for it was not added 
to parser_strings.cpp alongside pi, tau, etc. This a stand alone 'now' 
issue.

The other current return types from f_clock() are:

1 - Whole seconds to system wall clock since Y2K.
2 - Fractional days to system wall clock since Y2K (The povr 'now').
3 - Difference between machine epoch and y2k epoch in seconds.
4 - Convert local seconds since machine epoch to utc.

The second argument is whether to account for dst, or not, for types 1,2,3.

The third and last argument is the local seconds argument for return 
type utc. The method for this turned out uglier than hoped, though it 
appears to work.

I don't consider the form of f_clock final - in povr, as often as not, 
just trying outs ideas. A type to directly return seconds since machine 
epoch might be nice. Maybe other stuff. We'll see.

---
Putting more of the whole plan/implementation on the table because I see 
secondary 'concerns' such as POV-Ray's Y2K epoch not lining up with any 
unix like system I've ever used - all have been at Jan 1, 1970. This 
makes understanding the values POV-Ray is generating and using harder to 
work with.

Unix commands like date have some nice options:

date --date='TZ="America/Los_Angeles" 09:00 next Mon' +"%s"

but we cannot use the results directly in POV-Ray. Hmm, maybe an 
explicit unix time to y2k seconds type for f_clock...

---
I'll end with my view our 'now' is trying to be both a timer and a 
time/date input and I'm not sure these are completely compatible. Most 
of the time/date stuff which works for dst uses only seconds resolution 
as far as I know(1).

---
Aside:
Supposing datetime fixed to support %z %Z, and 'now' returning always a 
no dst utc or local time you can fairly easily adjust the fractional day 
value to account for dst of say an hour with:

#declare DSToffset = (1/24);

#debug concat("now + (1/24) ",
datetime(now+DSToffset,"%Y-%m-%d %H:%M:%S %z"),"\n")

#debug concat("now + (1/24) ",
datetime(now+DSToffset,"%Y-%m-%d %H:%M:%S %Z"),"\n")

FWIW.

Bill P.

(1) - The C++/Boost (plus time.h etc) date/time options go on forever. I 
don't begin to understand them all and it looks like c++20 is aiming to 
revamp/extend a lot of it. Coding wise. I tried to stick mostly with c.

Post a reply to this message

Am 11.08.2021 um 14:57 schrieb William F Pokorny:

> In your preamble, you didn't mention 'now' being used as an SDL timing 
> mechanism. It's something people do today and it carries risk with 
> respect to any particular timing result. The reasons is we are using the 
> system_clock for 'now' which can be updated at any time moving forward 
> or backward - synchronization with time servers, etc.
> 
> We don't warn against timing use of 'now' in the documentation as far as 
> I know. Perhaps we should?

I don't think there's much sense to warn of using `now` in this capacity 
in particular.

If people want a rough idea of how long the individual pieces of their 
scene take to parse, for example to decide what portions of their scene 
to try and optimize, `now` is probably good enough. It's not like system 
clocks are typically corrected umpteen times a day, or changed by huge 
values.

Also, given that `now` has days as its unit of measure, and the docs 
make no promises about its precision other than that it is "seconds or 
better", I think that's enough of a warning to not use it for 
microseconds precision measurements if anything serious depends on it.

If people want precise performance metrics, anything based on a real 
time clock - even `std::chrono::steady_clock` - is a poor choice of tool 
anyway. In general, I would guess that the task/thread scheduling jitter 
introduces far more noise into the measurements than the occasional 
system clock adjustment. CPU time would be a far better candidate for 
such things.


Also note that for the purpose of performance metrics, we already 
provide measurements of the total parsing time, both real time and CPU time.

If you think we should provide more tools for performance measurements, 
I would argue that some `#split_time` directive to print out the current 
parser stats would make more sense than some basic building block that 
users would have to build additional SDL code around, that in turn would 
potentially skew the results, to actually put to use.

But I don't see any dire need for such a feature.


> In povr's f_clock() the up front returned time comes from steady_clock 
> instead - which on my Ubuntu 20.04 machine appears to be a monotonic 
> clock since the last reboot/boot.

That's what `steady_clock` is for. If it wasn't monotonic, it would 
violate the C++ specs.

But being monotonic and steady, it is obviously no good as a building 
block for getting the time of day as a string (and conversion to an 
absolute date may also be a pain).


> Of note for POV-Ray v3.8 and onward is that the Fn_TimePt00, Fn_TimePt00 
> method for capturing time points should work in v3.8 beta 1, but it 
> doesn't because when 'now' was added, the support for it was not added 
> to parser_strings.cpp alongside pi, tau, etc. This a stand alone 'now' 
> issue.

Then that sounds like a bug to me, or at least an issue.


> Putting more of the whole plan/implementation on the table because I see 
> secondary 'concerns' such as POV-Ray's Y2K epoch not lining up with any 
> unix like system I've ever used - all have been at Jan 1, 1970. This 
> makes understanding the values POV-Ray is generating and using harder to 
> work with.

This was a deliberate choice, based on various considerations:


* The value was to be exposed to the user, allowing them to assign it to 
other variables, as well as perform arithmetics on it (such as adding a 
day or an hour before converting to a date/time string, or computing the 
number of minutes the parsing took and inserting that value as text in 
the image; or computing a value from scratch to be later represented as 
a date/time string).

This inevitably meant that the format would have to be based on a 
floating-point representation.

This in turn meant that the maximum possible absolute precision would 
not be constant, but vary depending on the distance from whatever "zero 
point" we would choose. So it would be advantageous to choose a "zero 
point" close to the time we live in.


* The value should have a precision of seconds in the worst case, but 
higher precision if possible.


* With POV-Ray being a platform-agnostic project at heart, I did not 
want to give preference for any existing platform-specific time 
representation format - neither POSIX' native seconds-since-1970 
timestamp format, nor Windows' native 100ns-intervals-since-1601 format. 
The only format I would have considered would have been a format defined 
as part of the C/C++ standard. Had I followed that route, I would have 
ended up using 1900 as the basis, as that's the only epoch year that had 
been explicitly specified by the official C/C++ standard at the time 
(see the `std::tm` structure). The POSIX epoch hasn't made it into the 
official C/C++ standard until C++20, and even then only for non-C 
features for now.

Also remember that back then (2013 or earlier), POSIX time wasn't nearly 
as ubiquitous as it is today.


* I did want a format that would be easy for users to understand, and

* I specificially did want a format that would be easily convertible to 
POSIX time.

While the latter would seem to clearly speak in favor of POSIX time 
itself, in conjunction with the previous point it actually speaks 
AGAINST it; here's why:

POSIX time is often cited as being the number of seconds that have 
elapsed since 1970-01-01 00:00 UTC.
That is NOT the case.
As a matter of fact, POSIX time is currently about 30 seconds short of 
that value.

Instead, POSIX time is the number of _days_ that have elapsed since that 
point in time, plus the number of seconds that have elapsed since 00:00 
UTC of the current (UTC) day.
(Modulo one or two other minor quirks.)
The difference being that this method does not include leap seconds.

By specifying the format of `now` to give a number of days, rather than 
seconds, we're avoiding this subtle difference between what peopole 
_think_ POSIX time is, and what it _actually_ is.
(As a minor snag, on days with leap seconds we're actually off by up to 
1 second in the value reported, but those days are rare.)


Now with using days instead of seconds for the `now` format, some math 
would be required to convert to or from POSIX time anyway; so 
introducing a constant offset in addition to the seconds-per-day factor 
was anything but a big deal.

Year 2000 made for a far nicer epoch than 1970, or 1961, or even 1900. 
And it was closer to "home", meaning we'd have more headroom in terms of 
precision for longer time in the future.

(Year 2000 also had the benefit that UTC was already defined in its 
current form then. In 1970 it wasn't, so the POSIX time value 0 is 
somewhat ambiguous. Not that it matters, as current dates are still 
well-defined.)


* Speaking of conversion, If it would turn out that users wanted to 
convert to or from POSIX time frequently, enough so that it would 
warrant support from our side, the format choisen would have made it 
trivial to provide an official include file to define a function or 
macro to convert between the two formats. And other then-competing 
formats besides.


> Unix commands like date have some nice options:
> 
> date --date='TZ="America/Los_Angeles" 09:00 next Mon' +"%s"
> 
> but we cannot use the results directly in POV-Ray. Hmm, maybe an 
> explicit unix time to y2k seconds type for f_clock...

What's wrong with a simple conversion function?

Why put all the burden on the building block that has the primary job of 
getting the current date and time in _some_ format?

That's a difficult enough job as it is. Leave conversion stuff to 
separate dedicated functions. It's also a difficult enough job.

There is _no_ reason whatsoever to mix them. Just as there is _no_ 
reason whatsoever to cram `now` and `datetime` in a single monolithic 
function, as the original design would have had it.


> Aside:
> Supposing datetime fixed to support %z %Z, and 'now' returning always a 
> no dst utc or local time you can fairly easily adjust the fractional day 
> value to account for dst of say an hour with:
> 
> #declare DSToffset = (1/24);

Yeah, no.
I'm already working on a solution that is far simpler for the user.


> (1) - The C++/Boost (plus time.h etc) date/time options go on forever. I 
> don't begin to understand them all and it looks like c++20 is aiming to 
> revamp/extend a lot of it. Coding wise. I tried to stick mostly with c.

Yeah, it makes for fun reading. Especially the parts where you go, 
"yeah, you got THIS functionality here, and THAT functionality there, 
but all of this doesn't give me THAT THING I REALLY WANT, which should 
be far more trivial to implement than THIS and THAT?! And probably just 
as common a use case."

Case in point:

There are two functions to convert `std::time_t` to a `std::tm` 
structure: One giving you the UTC representation, the other giving you 
the local time representation.

There is also a function to convert a `std::tm` in local time 
representation to `time_t`.

But where is the **** function that will allow me to populate a 
`std::tm` with a UTC representation and convert *THAT* to `time_t`? That 
would have to be even easier to implement!
So how else do I get a `time_t` representation of 2000-01-01 00:00 UTC?!

And no, just casting 30 years' worth of seconds to `time_t` is NOT the 
proper way to do it folks, because `time_t` is NOT guaranteed to use 
POSIX representation. Not even in C++20.


I've now decided that the cleanest way of doing it (in pure C++11 
without boost anyway) may be to convert 2000-01-01 00:00 local time to 
`time_t`, convert that into UTC, convert that back again to `time_t` 
pretending it to be local time, compute the difference to the first 
`time_t` obtained, and subtract the result from that first `time_t`.

Behold - 2000-01-01 00:00 UTC as `time_t`. Easy as pi. Or half a tau. 
Anyway, irrational. Transcendental even.

Post a reply to this message

Am 11.08.2021 um 14:57 schrieb William F Pokorny:

> Of note for POV-Ray v3.8 and onward is that the Fn_TimePt00, Fn_TimePt00 
> method for capturing time points should work in v3.8 beta 1, but it 
> doesn't because when 'now' was added, the support for it was not added 
> to parser_strings.cpp alongside pi, tau, etc. This a stand alone 'now' 
> issue.

I think you mean `parser_functions.cpp`?

There's a point to be made in keeping it that way: If we allow `now` in 
a function, what value should it evaluate to? The point of time when the 
function is parsed and compiled, or the one when it is invoked?

Since the answer to both alternatives is probably "yes" (depending on 
the use case), allowing it at all in functions _will_ cause confusion.

For e.g. `pi` and `tau`, by contrast, the answer is easy: "Who cares, 
it's the same either way."


Also, if you design a function to return current date and time encoded 
in a different time format (say, POSIX timestamp), it's easy to just add 
another parameter, and pass `now` when invoking that function. (*)

And while a function that evaluates to a different value upon each 
invocation during render _may_ sound like a fun idea, I'm quite sure its 
applicability is pretty much limited to being a novelty.


(*Speaking of extra parameters: Passing the number of seconds in a day 
as a _parameter_...??!)

Post a reply to this message

On 8/11/21 2:07 PM, clipka wrote:
> If people want precise performance metrics, anything based on a real 
> time clock - even `std::chrono::steady_clock` - is a poor choice of tool 
> anyway. In general, I would guess that the task/thread scheduling jitter 
> introduces far more noise into the measurements than the occasional 
> system clock adjustment. CPU time would be a far better candidate for 
> such things.
> 
> 
> Also note that for the purpose of performance metrics, we already 
> provide measurements of the total parsing time, both real time and CPU 
> time.
> 
> If you think we should provide more tools for performance measurements, 
> I would argue that some `#split_time` directive to print out the current 
> parser stats would make more sense than some basic building block that 
> users would have to build additional SDL code around, that in turn would 
> potentially skew the results, to actually put to use.
> 
> But I don't see any dire need for such a feature.

First, thanks for the detailed response.

I remain of the opinion POV-Ray would be better positioned with two 
'now's with a 'now_for_datetime' and a 'now_for_timing'.

A now_for_datetime can be sloppy to the order most concerns we've 
pointed out are OK. It could always use the local machine's epoch and 
local setting - aligning with other local tools in the machine/OS 
environment. Something easier for users to use date time wise.

As to timing things inside the parser. My initial reaction was similar 
to yours. The idea rattled around a while inside the cavity behind my 
eyes, and, I started to think that certain sorts of timing 
characterizations are likely simpler/better-done within the parser.

The parser, in being single threaded, is at some advantage timing wise 
for using only a one thread.

---
Something which gets asked every now and again, is whether POV-Ray could 
provide some idea as to the relative expense of certain features over 
others.

Instead of timing some block of user SDL code, imagine instead a parser 
only job which first characterizes the cost of an empty loop and - to 
the degree it can - other "wrapper" costs. We then start inserting 
things to characterize in that loop/wrapper. Single patterns in turn as 
called via functions. Functions themselves one at a time. Or trace() 
operations against base shapes/surfaces one at a time. Or particular 
parser functions one a time. Or whatever else. In many respects the 
parser can more easy isolate particular functionality than can be 
accomplished via other methods.

Imagine a large, long running, parser characterization job for all the 
'things' for which we want to track relative performance. The 
information from which we, perhaps, stick in a big table via jr's new 
macro and publish. Regular timing results useful to users and developers.

This the sort of parser timing application which I believe requires a 
more stable and dedicated 'now_for_timing'. But, yep, it's just another 
idea at this point - except for now having povr's timer specific 
f_clock() feature in hand.

Bill P.

Post a reply to this message

On 8/12/21 7:59 AM, clipka wrote:
> Am 11.08.2021 um 14:57 schrieb William F Pokorny:
> 
>> Of note for POV-Ray v3.8 and onward is that the Fn_TimePt00, 
>> Fn_TimePt00 method for capturing time points should work in v3.8 beta 
>> 1, but it doesn't because when 'now' was added, the support for it was 
>> not added to parser_strings.cpp alongside pi, tau, etc. This a stand 
>> alone 'now' issue.
> 
> I think you mean `parser_functions.cpp`?

I did, sorry.

As for what you say below... We're not going to agree. Plus, I need to 
run to other stuff for the day.

The token 'now' is what it is when the parse sees it. It's no different 
than 'pi' in that regard. Why create inconsistent behavior for users?

> 
> There's a point to be made in keeping it that way: If we allow `now` in 
> a function, what value should it evaluate to? The point of time when the 
> function is parsed and compiled, or the one when it is invoked?
> 
> Since the answer to both alternatives is probably "yes" (depending on 
> the use case), allowing it at all in functions _will_ cause confusion.
> 
> For e.g. `pi` and `tau`, by contrast, the answer is easy: "Who cares, 
> it's the same either way."
> 
> 
> Also, if you design a function to return current date and time encoded 
> in a different time format (say, POSIX timestamp), it's easy to just add 
> another parameter, and pass `now` when invoking that function. (*)
> 
> And while a function that evaluates to a different value upon each 
> invocation during render _may_ sound like a fun idea, I'm quite sure its 
> applicability is pretty much limited to being a novelty.
> 
> 
> (*Speaking of extra parameters: Passing the number of seconds in a day 
> as a _parameter_...??!)

Post a reply to this message

Am 12.08.2021 um 14:16 schrieb William F Pokorny:

> A now_for_datetime can be sloppy to the order most concerns we've 
> pointed out are OK. It could always use the local machine's epoch and 
> local setting - aligning with other local tools in the machine/OS 
> environment. Something easier for users to use date time wise.

Um - nope.

Juging from my experience with trying to get the `datetime` and `now` 
stuff sorted out, a `now` based on local time will only be a source of 
utter confusion.

What would that value even represent?

The only sensible formulation is a "time units since D-Day H-Hour".

Such a D-Day H-Hour would have well-defined DST semantics, either 
explicitly by definition or implicitly by user expectation. If D-Day was 
2000-01-01, H-Hour was 00:00, and the value was advertised as local 
time, then users would expect H-Hour to imply local non-DST in the 
northern hemisphere, local DST in the southern hemisphre, or non-DST if 
their time zone does not observe DST at all.

So if some user were to compute a point in time of, say, 2000-07-01 
00:00 local time - would that be D-Day H-Hour plus exactly 182*24 hours, 
or 1 hour more or less due to DST?


That won't fly. Not in the "easier for the user" sort of way, at any rate.

And certainly not in the "easier for the programmer" sort of way either, 
that much I can guarantee already.


> Imagine a large, long running, parser characterization job for all the 
> 'things' for which we want to track relative performance. The 
> information from which we, perhaps, stick in a big table via jr's new 
> macro and publish. Regular timing results useful to users and developers.

And that will be utterly useless if the functionality to actually 
implement the timing measurements takes considerable time compared to 
what we actually want to measure.

For such an endeavor, things need to have the lightest footprint in 
parser itself that we can possibly create. Ideally, it would just be a 
single (and possibly even very short) token. What that statement would 
trigger can be a bit more complex, because it will be compiled, rather 
than having to be parsed.

Possibly a job for a dedicated build, that adds a single-character token 
just for the purpose, and which isn't functional in release builds. "@" 
isn't used anywhere, for example.

This could trigger one or more performance metrics to be logged whenever 
"@" is encountered, and the result dumped once parsing (or rendering) 
has completed. Maybe a fixed number of characters to follow (say, 4), to 
be logged along with it. The resulting dump could then be correlated 
with the scene later.

If you want precise results, that would be the road to give you the most 
reliable data.

Post a reply to this message