Usually we can use extra tools to generate data set. Then visualize it in Tableau. This may decrease flexibility. Actually, we can do everything in Tableau natively: for both data set creation and visualization.  In this way, we may need to push Tableau's capability to the extreme and so be it. It's like running LINPACK to test a new computer and to access its performance.

Here I use Tableau alone to create data set and visualization. Using similar optimization techniques as before. But it is still quite slow to my taste.

Julia set provides us with a great variety of fractals which are wildly beautiful. You are welcome to download it, play with it and leave questions in the comment area. More initial parameters can be found in Wikipedia's Julia set entry for creating various Julia set images.
Conclusion and issues
While Tableau provides me with a wonderful graphical tool and I got great personal satisfaction in exploring the beauty of mathematics, I feel that fractals and math function rendering can be used to push the boundary of computation engine in Tableau: data structure, memory allocation, compiler, rendering. Maybe there is something to rethink of, because there seems much room for improvement in the simple computing of fractals.

The higher the number of points (in horizontal or vertical dimensions), and the higher the number of iterations, the better resolution of the image. But it will be really slow if the total number of pixels is greater than a few millions.

Most computation above uses a long string to store a state vector (or vector state?). It includes the x and y coordinates of each data mark. The manipulation of this long string seems to cost a lot of computation time.

Does introduction of local variables (registers) inside the formula editor help? Just a thought.

Using Tableau alone, one can already create a variety of fractal images. It maybe a little slow, but it's simple and a lot of fun. You can download the workbook here and find the initial c parameters in the above wiki link or over the web.

Note that try to start with just a few iterations and a small number of points. Otherwise it may take a long time. Also, disable the auto-update in worksheet or dashboard menu while you set up the parameters. Have fun!