Thielak
Developer
Posts: 14
Joined: Tue May 02, 2017 11:59 pm

Possible issue with random [SOLVED]

Thu Jun 01, 2017 12:13 am

While making a silly pattern that generates a name by putting together a random adjective from a list and a random name from another, I noticed a possible issue with Random.

More specifically, it seems to happen when it is used with a list of <Item> instead of the "chain" (a|b|c|d) notation.

Take this sample Model, that takes the user's input N and "rolls" a number from 1 to 6 N times. I know it's not an elegant approach but I'm using it to let you reproduce what's happening.

Code: Select all

		<Model>
			<Pattern>ROLL *</Pattern>
			<Response>
				<Think><Var Set="count">0</Var></Think>
				<Bind Key="{0}"><Match /></Bind>
				<While Var="count" Not="{0}">
					<Think><Var Set="count"><Math Get="increment"><Var Get="count" /></Math></Var></Think>
					<Random>
						<Item>1</Item>
						<Item>2</Item>
						<Item>3</Item>
						<Item>4</Item>
						<Item>5</Item>
						<Item>6</Item>
					</Random>
				</While>
			</Response>
		</Model>
With that Model I get long streaks of the same roll, and the same happens if I substitute the random block with <Math Get="random" Min="1" Max="6" />; on the other hand, if I substitute that random block with <Random>(1|2|3|4|5|6)</Random> the result looks fine.

I tried declaring the random object separately

Code: Select all

    <Random Name="roll">
        <Item>1</Item>
        <Item>2</Item>
        <Item>3</Item>
        <Item>4</Item>
        <Item>5</Item>
        <Item>6</Item>
    </Random>
calling it with <Random Get="roll" />, but I had the same results: fine if I use (1|2|3|4|5|6), not fine otherwise. When I tried making a pattern exclusively for rolling 1-6 once and called it with Gotos the problem looked less severe, but it was still there.

To me it looks like multiple random calls are using the same random seed, and that seed doesn't change fast enough.
The fact that with Gotos the problem looked less severe might support that thesis, as Goto is inherently slower, giving the seed a better chance to change between the multiple Random calls.

As for why the (a|b|c|d) notation seems better, I have no idea, other than you might be using a different way to pick a random entry from those because you don't have to account for the possibility of having to weight some entries differently (<Item Weight="x">).
Last edited by Thielak on Sat Jun 10, 2017 2:00 am, edited 1 time in total.

Archi
Expert
Posts: 19
Joined: Wed May 03, 2017 11:13 pm

Re: Possible issue with random

Thu Jun 01, 2017 1:44 am

I took a quick look and it looks like Random() generator is being initialized for each <Random> invocation, which can lead to the same seed calculated from current system clock several times in a row, and that leads to the same generated response on following calls, until clock changes. This leads to a responses such as 1 1 1 1 4 4 4 3 3 6 6 6 2 2 etc.

A solution would be probably to use either global random generator, or a random generator per each AI call, and not per each invocation. That would guarantee either even (enough) global distribution across all random calls, or even distribution across a single call.

User avatar
Fantom
Help & Support
Posts: 312
Joined: Fri Oct 25, 2013 9:20 pm

Re: Possible issue with random

Thu Jun 01, 2017 3:48 pm

@Thielak Does the behavior stay the same in your SIML Bot? I mean within your application or just Syn Bot Studio.

Syn Bot Studio is still running on version 2.0.1. Which is pretty old.

On my end the following test seems to be passing:

CSharp Code

[TestMethod]
public void RandomTag_Seeding()
{
	for (var i = 4; i < 8; i++)
	{
		//Can be replaced with "ROLL *" in your case
		var result = TestHelper.GetResult($"random seeding test {i}");
		var firstChar = result.BotMessage[0];
		var sequencedString = Sequence(firstChar, i);
		Assert.AreNotEqual(sequencedString, result.BotMessage);
	}
}

private static string Sequence(char a, int sequenceLength)
{
	var builder = new StringBuilder();
	for (var j = 0; j < sequenceLength; j++)
	{
		builder.Append(a);
	}
	return builder.ToString();
}

Archi
Expert
Posts: 19
Joined: Wed May 03, 2017 11:13 pm

Re: Possible issue with random

Thu Jun 01, 2017 11:19 pm

FranzZ wrote:@Thielak Does the behavior stay the same in your SIML Bot? I mean within your application or just Syn Bot Studio.

Syn Bot Studio is still running on version 2.0.1. Which is pretty old.

On my end the following test seems to be passing:

CSharp Code

[TestMethod]
public void RandomTag_Seeding()
{
	for (var i = 4; i < 8; i++)
	{
		//Can be replaced with "ROLL *" in your case
		var result = TestHelper.GetResult($"random seeding test {i}");
		var firstChar = result.BotMessage[0];
		var sequencedString = Sequence(firstChar, i);
		Assert.AreNotEqual(sequencedString, result.BotMessage);
	}
}

private static string Sequence(char a, int sequenceLength)
{
	var builder = new StringBuilder();
	for (var j = 0; j < sequenceLength; j++)
	{
		builder.Append(a);
	}
	return builder.ToString();
}
According to this code, it will check if bot responded with exactly the same content, not if the seed is different. Bot can respond with "a a a b" and that will not be equal to "a a a a" which will make test pass, yet this does not indicate that seed is not the same, because you have time between invocations and random seed generated on "new Random()" has around 20ms intervals.

In other words, if it takes longer than 20 ms for the bot to evaluate entire test, then this test will likely never raise issue (as the chance is too small), yet there will be the same seed used for several <Random> invocations.

User avatar
Fantom
Help & Support
Posts: 312
Joined: Fri Oct 25, 2013 9:20 pm

Re: Possible issue with random

Fri Jun 02, 2017 3:42 am

Sorry @Archi I didn't elaborate on my previous post. :( I wrote the test case to see if I could reproduce a specific pattern I saw in Syn Bot Studio.

Using the code provided by @Thielak, Syn Bot Studio gave the following outputs:

Input = Roll 2
Output = 55

Input = Roll 4
Output = 6666

Input = Roll 6
Output = 444444

Input = Roll 8
Output = 55555555

Seeing the above outputs, I decided to check if the same happened when I tried the code within my SIML based Bot application but It didn't.

Within my application the outputs were like:

Input = Roll 2
Output = 52

Input = Roll 4
Output = 4342

Input = Roll 6
Output = 135113

Input = Roll 8
Output = 45423515

I agree that it might not be the right test method but I just wanted to see if the pattern existed. It's nothing official.

I got in touch with the team to check the behavior and they concluded that the chain syntax uses a tailored utility class for random number generation (which was added in later versions of Syn.Bot framework) however this is not the case with <Item> elements within the <Random> tag. The team also added that they have their own "randomness" tests and they all seem to pass because the complexity and the environment of the tests provides enough time for the seed to change.

The dev team has already addressed this after being notified of your post. So you can expect it to be fixed in Syn.Bot v2.1.0 ;)

Thielak
Developer
Posts: 14
Joined: Tue May 02, 2017 11:59 pm

Re: Possible issue with random

Fri Jun 02, 2017 7:54 pm

FranzZ wrote:I got in touch with the team to check the behavior and they concluded that the chain syntax uses a tailored utility class for random number generation (which was added in later versions of Syn.Bot framework) however this is not the case with <Item> elements within the <Random> tag. The team also added that they have their own "randomness" tests and they all seem to pass because the complexity and the environment of the tests provides enough time for the seed to change.

The dev team has already addressed this after being notified of your post. So you can expect it to be fixed in Syn.Bot v2.1.0 ;)
Thank you for your interest in investigating the issue and for checking with the dev team about it.
I saw a new beta (2.1.0-beta-004) popped up earlier today, but that doesn't include the fix for this issue. Is the change expected to be in a future beta update or will it be shipped only in the final 2.1.0 version?

User avatar
Fantom
Help & Support
Posts: 312
Joined: Fri Oct 25, 2013 9:20 pm

Re: Possible issue with random

Fri Jun 02, 2017 9:11 pm

Though I am not sure, I have just uploaded the last compiled version by the dev team as v2.1.0-beta-005. I am yet to receive the status of the report I sent. If its not troublesome could you please provide me a sample code to test within a console application? I am unable to reproduce this here inside my test Bot application though in Syn Bot Studio it clearly seems to be using the same seed for <Item> elements within <Random>.

Thielak
Developer
Posts: 14
Joined: Tue May 02, 2017 11:59 pm

Re: Possible issue with random

Sat Jun 03, 2017 3:43 am

I was accidentally doing my second test on beta-003. beta-004 and beta-005 seem to have indeed solved the issue. I'm sorry for the inconvenience and thank you again!

User avatar
Fantom
Help & Support
Posts: 312
Joined: Fri Oct 25, 2013 9:20 pm

Re: Possible issue with random

Sat Jun 03, 2017 5:33 am

well thats awesome :) Thank you! for reporting back.

Thielak
Developer
Posts: 14
Joined: Tue May 02, 2017 11:59 pm

Re: Possible issue with random [SOLVED]

Sat Jun 10, 2017 2:00 am

I hadn't checked all of the possible options I explained in my first post and it looks like random in Math Get still exhibits the old behaviour even on the latest version. A SIML like this one:

Code: Select all

<?xml version="1.0" encoding="utf-8"?>
<Siml xmlns:Think="http://syn.co.in/2014/siml#think">
	<Concept Name="_mathdebug">
		<Model>
			<Pattern>MATHROLL</Pattern>
			<Response>
				<Var Think:Set="count" Value="0" />
				<While Var="count" Not="50"><Var Think:Set="count"><Math Get="increment"><Var Get="count" /></Math></Var><Math Get="random" Min="1" Max="6" /> </While>
			</Response>
		</Model>
	</Concept>
</Siml>
Will return results grouped together in such a way that it's clear some of them are using the same seed

Return to “Report Bugs and Issues”

Who is online

Users browsing this forum: No registered users and 1 guest