2014-01-16

Sample Last N Features

(FME 2014 build 14230)

Congrats FME 2014 release!

The Sampler transformer has "First N Features" option as "Sampling Type" parameter.
Well, is it possible to sample "Last N Features"?

First inspiration is: to reverse the order of input features using a Sorter, and then apply a Sampler with "First N Features" option.
It's easy, but I'm afraid that it would consume huge memory especially when there are very many features, because the Sorter will store all input features while processing. And also, if the input order of features has to be preserved, it will have to be reversed again after sampling.

Creating a custom transformer using an AttributeCreator with "Multiple Feature Attribute Support" option could be a workaround.







"SAMPLING_AMOUNT" is a Published Parameter defined in the custom transformer.
Since the back ground of "Value" setting which used the parameter with the manner as shown is colored by pink, it might be an unexpected usage. But in my test, it worked fine without any error or warning. Sometimes an unexpected usage is also effective.

Test and Result:







-----
# LastNSampler PythonCaller Edition
# Using Queue module
import fmeobjects, Queue

class LastNSampler(object):
    def __init__(self):
        # Create a FIFO (First In, First Out) queue.
        self.q = Queue.Queue(int(FME_MacroValues['SAMPLING_AMOUNT']))
     
    def pop(self, flag): # flag is a value indicating sampled or not.
        feature = self.q.get()
        feature.setAttribute('_sampled', flag)
        self.pyoutput(feature)
     
    def input(self, feature):
        if self.q.full():
            self.pop('no')
        self.q.put(feature)
     
    def close(self):
        while not self.q.empty():
            self.pop('yes')
-----
For what it's worth...

No comments:

Post a Comment