[kepler-users] Size Limitation of ArrayTokens?

[Stefan Proell]

Hi!
There seems to be a problem with the buffer in the BinaryFileReader 
Class, which is limited to 20000 bytes, as you can see in the 
corresponding source file (line 277).

     /** The current bytes read. */
     protected byte[] bytesRead = new byte[20000];

It only is reading this first 20000 bytes and does not proceed after 
this. I will have a closer look at ...

Stefan

Am 2012-05-22 15:12, schrieb Stefan Proell:
> Dear Matt,
> thanks for your example. It works fine, but does not solve my problem. 
> I use a Binary File Reader to read a file from my local disk. The 
> Binary File Reader reads a local file path or URL and outputs an array 
> of bytes. No matter what file I want to read, the array size is 
> exactly 20 000 (bytes). I attached a small sample workflow to 
> demonstrate this. It simply reads a file (in my testcase this file is 
> about 40 MB) and displays the length of the resulting Byte Array.
>
> Do you have any idea why this behaviour occurs and can your reproduce it?
>
> Kind regards,
> Stefan
>
>
> Am 2012-05-21 19:05, schrieb Matt Jones:
>> I also took a look, and tried a larger array -- the attached workflow 
>> processes 30K integers through an array and back out again, and works 
>> fine.   So there isn't an inherent limit at 20K elements.  So I'm not 
>> quite sure what's happening with yours, but maybe this will help to 
>> debug.
>>
>> Matt
>>
>>
>> On Mon, May 21, 2012 at 8:32 AM, Christopher Brooks 
>> <cxh at eecs.berkeley.edu <mailto:cxh at eecs.berkeley.edu>> wrote:
>>
>>     HI Stefan,
>>     A quick search of the web shows no apparent limits.
>>     A more complete test case would help.
>>
>>     The way I would approach this is by splitting up the problem into
>>     first just reading the data and being sure that worked and then
>>     adding more like encoding.
>>
>>     _Christopher
>>
>>
>>     On 5/21/12 9:20 AM, Stefan Proell wrote:
>>
>>         Dear Mailing list,
>>         I wrote my own Actor which I need in order to encode binary
>>         files via Base64 and pass the resulting encoded String to a
>>         REST Service. I feed the binary file with a
>>         BinaryFileReader-Actor to my encoding Actor and cast the Token
>>         to an Array, as I found no other solution to receive the bytes
>>         from the file. I then use a standard (and also deprecated)
>>         method for encoding the file first to Base64 and then send it
>>         through some URL-safe encoding. The method looks like this:
>>
>>
>>         @Override
>>            public void fire() throws IllegalActionException {
>>
>>                super.fire();
>>
>>                // Read file from InputPort  and convert to a ByteArray
>>                ArrayToken inputToken = (ArrayToken) inputPort.get(0);
>>                byte[] inputBytes =
>>         ArrayToken.arrayTokenToUnsignedByteArray(inputToken);
>>
>>                // Encoding
>>                String encode = new
>>          String(Base64.encodeBase64(inputBytes));
>>                String encodedString = 
>> java.net.URLEncoder.encode(encode);
>>
>>                output.send(0, new StringToken(encodedString));
>>
>>            }
>>
>>         My problem is that the encoded file is truncated by the actor
>>         and hence not usable for further processing. There seems to be
>>         a limit of 20 000 array elements (bytes), which is not
>>         sufficient for my purpose. Does anyone have an idea why the
>>         file is chopped off after 20k letters?
>>
>>         Thanks in advance,
>>         Stefan
>>
>>         _______________________________________________
>>         Kepler-users mailing list
>>         Kepler-users at kepler-project.org
>> <mailto:Kepler-users at kepler-project.org>
>>         http://lists.nceas.ucsb.edu/kepler/mailman/listinfo/kepler-users
>>
>>
>>     --     Christopher Brooks, PMP                       University 
>> of California
>>     CHESS Executive Director                      US Mail: 337 Cory Hall
>>     Programmer/Analyst CHESS/Ptolemy/Trust        Berkeley, CA 
>> 94720-1774
>>     ph: 510.643.9841 <tel:510.643.9841>                              
>>      (Office: 545Q Cory)
>>     home: (F-Tu) 707.665.0131 <tel:707.665.0131> cell: 707.332.0670
>> <tel:707.332.0670>
>>
>>
>>     _______________________________________________
>>     Kepler-users mailing list
>>     Kepler-users at kepler-project.org
>> <mailto:Kepler-users at kepler-project.org>
>>     http://lists.nceas.ucsb.edu/kepler/mailman/listinfo/kepler-users
>>
>>
>
>
>
> _______________________________________________
> Kepler-users mailing list
> Kepler-users at kepler-project.org
> http://lists.nceas.ucsb.edu/kepler/mailman/listinfo/kepler-users