Test Case Expected Results

When creating Test Cases, capture expected results (in addition to capturing the name and test conditions as Lamatic does currently). Consider also, Using code checks and AI to find deviation and grade result quality. Use this quality score as an input for deciding whether to deploy an update. Assist in recommending the best option when A|B testing prompts, agents, workflows, models and other settings such as temperature. Using these results to create benchmarks. Measuring and alerting about “model drift” over time. Integrating “human-in-the-loop” strategies to expand Test Cases and quality checking. Added Later (based on feedback from another user): Make it easier to see the history of Test Case executions (without leaving Experiments). Marc Greenberg didn’t realize he could find the results in Logs, but also didn’t like having to dig through logs to find the execution details for a given test.

cwhiteman 2 months ago

💡 Feature Requests

Deactivate individual widgets?

It might be helpful if there was a way to deactivate individual widgets, but leave them staged in the workflow, but turn them off while I’m experimenting with other configurations. For instance and the moment I have a code widget “Flatten array”, which is not connected in the workflow, but it seems to be breaking the API call even though it isn’t connected. My widget here with a bit of code is probably just rubbish and I will probably delete it in the end, But it would be nice if I could just deactivate it while I troubleshoot the other parts of the workflow, so I don’t need to delete it to test the other widgets. My apologies, maybe there is a way to do this already that I’m not aware.

celso.wilkinson About 2 months ago

1

💡 Feature Requests

Variable Selector intermittently fails to display some of the nodes

The variable selector only showed the first of many nodes. Occurred multiple times during this onboarding. Reloading the workflow would eventually correct the problem. Seems like a synchronization issue (are there multiple instances of the workflow or variable selector that can get out-of-sync?). See video at 58:40 if helpful to see it.

cwhiteman 11 days ago

1

🐛 Report Bug

Save Warning (when user is about to lose changes)

Customer lost a bunch of workflow work when he navigated away from the workflow (to reload the newly added Google Drive credential in the Google Drive Trigger Node) without first saving. Could solve either with AutoSave OR by warning the user when navigating away from a workflow with unsaved changes. If helpful, see onboarding video at 54:15.

cwhiteman 11 days ago

1

High Priority

👏🏻 Feedback

Hard to edit field when variable is present

if variable is present you cannot really edit the field or go to the field end. When you are clicking the field the variable selector pops every single time. Check on how other product solves this.

Aman Sharma 14 days ago

1

👏🏻 Feedback

LLM Credential Naming in the Index to DB node is confusing to users

The way we present the name of the model is confusing here (and perhaps elsewhere).

cwhiteman About 1 month ago

1

UI/UX

👏🏻 Feedback

On failure of any node return the exact point of failure

aruna About 2 months ago

💡 Feature Requests

Primary Key Selection in Indexing node

Primary Key not understandble - make this selectable from metadata fields instead

Aman Sharma About 2 months ago

1

High Priority

💡 Feature Requests

Model Preference & Fallback

A dedicated UI to choose model preferences on project level. There will be different categories of models ( Tabs ) for Text | Embedding| Chat | Image | Audio | Transcript For Each category customer can create 5 Settings and reorder them. In each settings can choose 4 tabs ( Basic | Fallback | Best | A/B Test ) When using model selection in a workflow node, choose a preferred setting options which will populate the UI with credentials and model selection. ( By Default the first preference will be chosen or create a new preferences

Aman Sharma 2 months ago

💡 Feature Requests

Pre populate model credentials Name

Nobody knows how to name the LLM. Can we populate the name that appears in the Model column (col 1) and let them overwrite it if they want?

Aman Sharma 2 months ago

👏🏻 Feedback

Improvement to Model Benchmark Screen

Take this Prototype for inspiration - https://cristi5833.softr.app/new Keep the Column Header Fixed Add Sort by in the headers itself for all columns Add Context Length Column Add Shortcut to model API screen ( Open Porta )

Aman Sharma 2 months ago

👏🏻 Feedback

Web Crawl Node

Create a Web Crawl Node to allow Lamatic to ingest and store data from a website to be accessed downstream. Specific requirements: 1) Accept a URL, multiple URLs, a sitemap.xml, or a URL pattern regex as a parameter to direct the crawl. Consider what would be needed to accept login credentials as a param to enable ingesting content behind a login (future). 2) Support the ability to store important metadata along with the visible content (for instance, the hreflang tag that will indicate the language & geography of the target audience). Consider a default metadata capture config that can be optionally reduced or expanded (future). 3) Generate status and error messaging: Accept parameters that configure status notifications (what notifications to send & to whom?). Thoughts/Questions: How to tell the Node where to store the crawled content? How should the visibility of this store be determined (in other words, is this store available only to this workflow, others within the Project, or others within the Organization)? Should the Node support an option to simultaneously vectorize the content as part of the crawl (to reduce storage needs)? Can/should we use the attribute on the Sitemap.xml to trigger this Node? Can/should the Node capture Image, Audio, and/or Video content? How should notifications & error handling work (regarding the status of the crawl)?

cwhiteman 2 months ago

1

💡 Feature Requests

Template Library

View different templates in one Place

Aman Sharma 4 months ago

1

High Priority

💡 Feature Requests

Exposing LLM API Keys

As I recall we used to hide API keys after they’ve been input (to give the CTO confidence that the company’s key won’t be compromised by Lamatic). Today I noticed that this is no longer true (I’m an Editor in this customer project and I can see his key):

cwhiteman 1 day ago

🐛 Report Bug

Index Node: variable selection bug

“Vectors”: when using the + icon to select variable, user must manually add “.vectors” to the end. “Metadata”: when using the + icon to select variable, user must manually add “.metadata” to the end. Should make this happen automatically. Also, it should be easier to see & update what’s in these text boxes.

cwhiteman 1 day ago

🐛 Report Bug

Vectorize Node: variable selection bug

“Texts to Vectorize”: when using the + icon to select variable, user must manually add “.content” to the end. Should make this happen automatically. Also, it should be easier to see & update what’s in this text box.

cwhiteman 1 day ago

🐛 Report Bug

Show logs in Realtime

Support logs fetch in realtime like cloudflare

aruna 3 days ago

💡 Feature Requests

Refresh Google Drive Trigger Folder property (after adding a new credential)

See 40:00 during onboarding, customer added a Google Drive folder via the Workflow builder. To see the new folder, customer had to exit the workflow and re-enter it. Can we trigger this refresh instead?

cwhiteman 11 days ago

👏🏻 Feedback

Add Gemini embedding model

Customer tried to set up a RAG project using Gemini, but there were no embedding models to select.

cwhiteman 11 days ago

💡 Feature Requests

Variable Selector: order nodes sequentially

Order the nodes sequentially when selecting variables from the + (button). It’s hard to find the node you want when they aren’t in the same order as the workflow:

cwhiteman 11 days ago

👏🏻 Feedback