Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [X...X.......................................................] duration: 6.462833816s FAIL (2025-10-15 10:31:02 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, Piece ID: VH7ZJKSZ2NUXQJZE2IKNWP5PAL3TN2NNHD3ATZUQHGKKYMSSM3LQ) context deadline exceeded; piecestore: (Node ID: 1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N, Piece ID: UMQRQYPZ7SWUCO2N7JDCEVI5U5P36FDL3SUM6HIYJVWVOZTWNXQQ) context deadline exceeded ======LOGS====== Oct 15 09:48:38.801058027 (524h17m24.26315051s) [upload=461218]: Starting upload Oct 15 09:48:38.801138119 (524h17m24.263230592s) [upload=461218]: Done starting upload Oct 15 09:48:38.801288545 (524h17m24.263381028s) [upload=461218]: Uploading segments... Oct 15 09:48:38.801442509 (524h17m24.263535002s) [upload=461218]: Got next segment. Inline: false Oct 15 09:48:38.801486663 (524h17m24.263579136s) [upload=461218]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 15 09:48:38.801493245 (524h17m24.263585728s) [upload=461218]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 15 09:48:39.066131918 (524h17m24.528224401s) [seg_pos={0 0}, upload=461218]: Begin upload segment... Oct 15 09:48:39.066205549 (524h17m24.528298022s) [seg_pos={0 0}, upload=461218]: Done begin upload segment. Oct 15 09:48:39.066243090 (524h17m24.528335573s) [noise=true , address=45.91.116.5:4299 , piece_id=4ARMXXE5LBRE2OZQEIITELKPTW4TISZWTD6XZGI4UJV7UCLUXAEQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.066254431 (524h17m24.528346915s) [noise=true , address=45.91.116.11:4117 , piece_id=VH7ZJKSZ2NUXQJZE2IKNWP5PAL3TN2NNHD3ATZUQHGKKYMSSM3LQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.066874873 (524h17m24.528967356s) [noise=true , address=145.79.227.7:4294 , piece_id=H4ZNVIXORNO4MTXOVYDQZI4IJE4VXHZWC6JI35YKOR3UESSXJNTQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.067301676 (524h17m24.529394159s) [noise=true , address=45.91.116.9:4262 , piece_id=X53JZGRLZLX53UZLLCXXUQCFSQD2FO76X74OM45G26HNDU55UXXQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.067705735 (524h17m24.529798218s) [noise=true , address=145.79.227.4:4187 , piece_id=TQVPL6ZA6LHKAJVL6XWSYASZD3MCKKV3553CRFD72XWTVWY62STQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.069183369 (524h17m24.531275862s) [noise=true , address=145.79.227.5:4222 , piece_id=C2JSPN2QVSGSXG6ZW3ZMOFX5YFNN65XY7E62N5OEHOOMK3NSO4KQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.070534782 (524h17m24.532627265s) [noise=true , address=77.90.31.11:4555 , piece_id=U5RYZSD643KSUI4XLL26EAN6FQRPMG62GQMMCQRPEEG2XKSNBIZQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.072356982 (524h17m24.534449466s) [noise=true , address=145.79.227.11:4730 , piece_id=VO2DSM5LT43QAFPWOY3W3PVEJ44GZJWTCWG5QRQQFB363DFVFZDA, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.073961167 (524h17m24.53605364s) [noise=true , address=45.91.116.4:4940 , piece_id=3MCSUDYN73NIVJTGKPOVY6N3MALB3GBXI7I5NAX4HXPDHVMQ52BA, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.075716960 (524h17m24.537809433s) [noise=true , address=145.79.227.16:4910 , piece_id=BIHEJDH7RPI4UCPUNV453QHSEVCLA54PSNNQB2II37XZDRXFZNWQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.079266419 (524h17m24.541358902s) [noise=true , address=145.79.227.14:4838 , piece_id=6K6A4CNEHGB3X6DEBLF5KHLUT4GE6BQBW6IE3RHBML2NWPN255AA, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.080943032 (524h17m24.543035505s) [noise=true , address=45.91.116.12:4226 , piece_id=DRXRGO7OXEKVS5ACL6FC7RKLGBBD6KNI7X6E5DM5D5VV5PEKN7FQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.081340479 (524h17m24.543432962s) [noise=true , address=45.91.116.8:4045 , piece_id=EDXBNLKKRZMT6MKK2KOIZM7DDKA3IHB6UMCXX4YKZF7JY2T3S5CQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.083616223 (524h17m24.545708706s) [noise=true , address=145.79.227.14:4840 , piece_id=B4ZVPRMVH7Y3TCWCLPO7DWRVMJL7VSI73CIT4PEHCMTIEURQKB2Q, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.083633946 (524h17m24.545726429s) [noise=true , address=45.91.116.14:4189 , piece_id=UMQRQYPZ7SWUCO2N7JDCEVI5U5P36FDL3SUM6HIYJVWVOZTWNXQQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.085493247 (524h17m24.54758573s) [noise=true , address=145.79.227.17:4969 , piece_id=R5HIP33QW5HLICARFWDUOW7I75YTDHZTJCIML5X52VKHLPL3J4YQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:40.727263388 (524h17m26.189355871s) [upload=461218]: Next returned nil segment Oct 15 09:48:40.727310076 (524h17m26.189402549s) [upload=461218]: Waiting for error group managing segments... Oct 15 09:48:41.004514306 (524h17m26.466606789s) [noise=true , address=145.79.227.7:4294 , piece_id=H4ZNVIXORNO4MTXOVYDQZI4IJE4VXHZWC6JI35YKOR3UESSXJNTQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.006116557 (524h17m26.46820904s) [noise=true , address=145.79.227.5:4222 , piece_id=C2JSPN2QVSGSXG6ZW3ZMOFX5YFNN65XY7E62N5OEHOOMK3NSO4KQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.012963184 (524h17m26.475055667s) [noise=true , address=145.79.227.16:4910 , piece_id=BIHEJDH7RPI4UCPUNV453QHSEVCLA54PSNNQB2II37XZDRXFZNWQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.012994684 (524h17m26.475087167s) [noise=true , address=145.79.227.14:4840 , piece_id=B4ZVPRMVH7Y3TCWCLPO7DWRVMJL7VSI73CIT4PEHCMTIEURQKB2Q, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.016649956 (524h17m26.478742439s) [noise=true , address=145.79.227.11:4730 , piece_id=VO2DSM5LT43QAFPWOY3W3PVEJ44GZJWTCWG5QRQQFB363DFVFZDA, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.019248864 (524h17m26.481341347s) [noise=true , address=145.79.227.17:4969 , piece_id=R5HIP33QW5HLICARFWDUOW7I75YTDHZTJCIML5X52VKHLPL3J4YQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.075991585 (524h17m26.538084068s) [noise=true , address=145.79.227.4:4187 , piece_id=TQVPL6ZA6LHKAJVL6XWSYASZD3MCKKV3553CRFD72XWTVWY62STQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.118655146 (524h17m26.580747619s) [noise=true , address=145.79.227.14:4838 , piece_id=6K6A4CNEHGB3X6DEBLF5KHLUT4GE6BQBW6IE3RHBML2NWPN255AA, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.285934795 (524h17m26.748027278s) [noise=true , address=77.90.31.11:4555 , piece_id=U5RYZSD643KSUI4XLL26EAN6FQRPMG62GQMMCQRPEEG2XKSNBIZQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.744525465 (524h17m27.206617948s) [noise=true , address=45.91.116.9:4262 , piece_id=X53JZGRLZLX53UZLLCXXUQCFSQD2FO76X74OM45G26HNDU55UXXQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:42.218961989 (524h17m27.681054462s) [noise=true , address=45.91.116.11:4117 , piece_id=VH7ZJKSZ2NUXQJZE2IKNWP5PAL3TN2NNHD3ATZUQHGKKYMSSM3LQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:42.313638675 (524h17m27.775731158s) [noise=true , address=45.91.116.14:4189 , piece_id=UMQRQYPZ7SWUCO2N7JDCEVI5U5P36FDL3SUM6HIYJVWVOZTWNXQQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:42.434283559 (524h17m27.896376053s) [noise=true , address=45.91.116.8:4045 , piece_id=EDXBNLKKRZMT6MKK2KOIZM7DDKA3IHB6UMCXX4YKZF7JY2T3S5CQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:42.434321832 (524h17m27.896414305s) [seg_pos={0 0}, upload=461218]: Segment reached optimal threshold of 13 pieces. Oct 15 09:48:42.434605041 (524h17m27.896697534s) [noise=true , address=45.91.116.5:4299 , piece_id=4ARMXXE5LBRE2OZQEIITELKPTW4TISZWTD6XZGI4UJV7UCLUXAEQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX): protocol: expected piece hash; context canceled; EOF Oct 15 09:48:42.434666138 (524h17m27.896758621s) [noise=true , address=45.91.116.5:4299 , piece_id=4ARMXXE5LBRE2OZQEIITELKPTW4TISZWTD6XZGI4UJV7UCLUXAEQ, seg_pos={0 0}, upload=461218]: Long tail cancellation due to optimal threshold Oct 15 09:48:42.434692849 (524h17m27.896785432s) [noise=true , address=45.91.116.12:4226 , piece_id=DRXRGO7OXEKVS5ACL6FC7RKLGBBD6KNI7X6E5DM5D5VV5PEKN7FQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1): protocol: expected piece hash; context canceled; EOF Oct 15 09:48:42.434706184 (524h17m27.896798657s) [noise=true , address=45.91.116.12:4226 , piece_id=DRXRGO7OXEKVS5ACL6FC7RKLGBBD6KNI7X6E5DM5D5VV5PEKN7FQ, seg_pos={0 0}, upload=461218]: Long tail cancellation due to optimal threshold Oct 15 09:48:42.434718318 (524h17m27.896810911s) [noise=true , address=45.91.116.4:4940 , piece_id=3MCSUDYN73NIVJTGKPOVY6N3MALB3GBXI7I5NAX4HXPDHVMQ52BA, seg_pos={0 0}, upload=461218]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe): protocol: expected piece hash; context canceled; EOF Oct 15 09:48:42.434728447 (524h17m27.89682093s) [noise=true , address=45.91.116.4:4940 , piece_id=3MCSUDYN73NIVJTGKPOVY6N3MALB3GBXI7I5NAX4HXPDHVMQ52BA, seg_pos={0 0}, upload=461218]: Long tail cancellation due to optimal threshold Oct 15 09:48:42.434760458 (524h17m27.896852931s) [seg_pos={0 0}, upload=461218]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 15 09:48:42.434866900 (524h17m27.896959384s) [upload=461218]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 15 09:48:42.434873884 (524h17m27.896966367s) [upload=461218]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 15 09:48:42.760633614 (524h17m28.222726087s) [upload=461218]: Done uploading segments... Oct 15 09:48:42.760655435 (524h17m28.222747918s) [upload=461218]: Upload finished. err: <nil> Duration: 6.462874654s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.X...X......................................................] duration: 6.392889715s FAIL (2025-10-15 10:30:17 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, Piece ID: VH7ZJKSZ2NUXQJZE2IKNWP5PAL3TN2NNHD3ATZUQHGKKYMSSM3LQ) context deadline exceeded; piecestore: (Node ID: 1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N, Piece ID: UMQRQYPZ7SWUCO2N7JDCEVI5U5P36FDL3SUM6HIYJVWVOZTWNXQQ) context deadline exceeded ======LOGS====== Oct 15 09:48:38.801058027 (524h17m24.26315051s) [upload=461218]: Starting upload Oct 15 09:48:38.801138119 (524h17m24.263230592s) [upload=461218]: Done starting upload Oct 15 09:48:38.801288545 (524h17m24.263381028s) [upload=461218]: Uploading segments... Oct 15 09:48:38.801442509 (524h17m24.263535002s) [upload=461218]: Got next segment. Inline: false Oct 15 09:48:38.801486663 (524h17m24.263579136s) [upload=461218]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 15 09:48:38.801493245 (524h17m24.263585728s) [upload=461218]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 15 09:48:39.066131918 (524h17m24.528224401s) [seg_pos={0 0}, upload=461218]: Begin upload segment... Oct 15 09:48:39.066205549 (524h17m24.528298022s) [seg_pos={0 0}, upload=461218]: Done begin upload segment. Oct 15 09:48:39.066243090 (524h17m24.528335573s) [noise=true , address=45.91.116.5:4299 , piece_id=4ARMXXE5LBRE2OZQEIITELKPTW4TISZWTD6XZGI4UJV7UCLUXAEQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.066254431 (524h17m24.528346915s) [noise=true , address=45.91.116.11:4117 , piece_id=VH7ZJKSZ2NUXQJZE2IKNWP5PAL3TN2NNHD3ATZUQHGKKYMSSM3LQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.066874873 (524h17m24.528967356s) [noise=true , address=145.79.227.7:4294 , piece_id=H4ZNVIXORNO4MTXOVYDQZI4IJE4VXHZWC6JI35YKOR3UESSXJNTQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.067301676 (524h17m24.529394159s) [noise=true , address=45.91.116.9:4262 , piece_id=X53JZGRLZLX53UZLLCXXUQCFSQD2FO76X74OM45G26HNDU55UXXQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.067705735 (524h17m24.529798218s) [noise=true , address=145.79.227.4:4187 , piece_id=TQVPL6ZA6LHKAJVL6XWSYASZD3MCKKV3553CRFD72XWTVWY62STQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.069183369 (524h17m24.531275862s) [noise=true , address=145.79.227.5:4222 , piece_id=C2JSPN2QVSGSXG6ZW3ZMOFX5YFNN65XY7E62N5OEHOOMK3NSO4KQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.070534782 (524h17m24.532627265s) [noise=true , address=77.90.31.11:4555 , piece_id=U5RYZSD643KSUI4XLL26EAN6FQRPMG62GQMMCQRPEEG2XKSNBIZQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.072356982 (524h17m24.534449466s) [noise=true , address=145.79.227.11:4730 , piece_id=VO2DSM5LT43QAFPWOY3W3PVEJ44GZJWTCWG5QRQQFB363DFVFZDA, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.073961167 (524h17m24.53605364s) [noise=true , address=45.91.116.4:4940 , piece_id=3MCSUDYN73NIVJTGKPOVY6N3MALB3GBXI7I5NAX4HXPDHVMQ52BA, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.075716960 (524h17m24.537809433s) [noise=true , address=145.79.227.16:4910 , piece_id=BIHEJDH7RPI4UCPUNV453QHSEVCLA54PSNNQB2II37XZDRXFZNWQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.079266419 (524h17m24.541358902s) [noise=true , address=145.79.227.14:4838 , piece_id=6K6A4CNEHGB3X6DEBLF5KHLUT4GE6BQBW6IE3RHBML2NWPN255AA, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.080943032 (524h17m24.543035505s) [noise=true , address=45.91.116.12:4226 , piece_id=DRXRGO7OXEKVS5ACL6FC7RKLGBBD6KNI7X6E5DM5D5VV5PEKN7FQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.081340479 (524h17m24.543432962s) [noise=true , address=45.91.116.8:4045 , piece_id=EDXBNLKKRZMT6MKK2KOIZM7DDKA3IHB6UMCXX4YKZF7JY2T3S5CQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.083616223 (524h17m24.545708706s) [noise=true , address=145.79.227.14:4840 , piece_id=B4ZVPRMVH7Y3TCWCLPO7DWRVMJL7VSI73CIT4PEHCMTIEURQKB2Q, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.083633946 (524h17m24.545726429s) [noise=true , address=45.91.116.14:4189 , piece_id=UMQRQYPZ7SWUCO2N7JDCEVI5U5P36FDL3SUM6HIYJVWVOZTWNXQQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.085493247 (524h17m24.54758573s) [noise=true , address=145.79.227.17:4969 , piece_id=R5HIP33QW5HLICARFWDUOW7I75YTDHZTJCIML5X52VKHLPL3J4YQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:40.727263388 (524h17m26.189355871s) [upload=461218]: Next returned nil segment Oct 15 09:48:40.727310076 (524h17m26.189402549s) [upload=461218]: Waiting for error group managing segments... Oct 15 09:48:41.004514306 (524h17m26.466606789s) [noise=true , address=145.79.227.7:4294 , piece_id=H4ZNVIXORNO4MTXOVYDQZI4IJE4VXHZWC6JI35YKOR3UESSXJNTQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.006116557 (524h17m26.46820904s) [noise=true , address=145.79.227.5:4222 , piece_id=C2JSPN2QVSGSXG6ZW3ZMOFX5YFNN65XY7E62N5OEHOOMK3NSO4KQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.012963184 (524h17m26.475055667s) [noise=true , address=145.79.227.16:4910 , piece_id=BIHEJDH7RPI4UCPUNV453QHSEVCLA54PSNNQB2II37XZDRXFZNWQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.012994684 (524h17m26.475087167s) [noise=true , address=145.79.227.14:4840 , piece_id=B4ZVPRMVH7Y3TCWCLPO7DWRVMJL7VSI73CIT4PEHCMTIEURQKB2Q, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.016649956 (524h17m26.478742439s) [noise=true , address=145.79.227.11:4730 , piece_id=VO2DSM5LT43QAFPWOY3W3PVEJ44GZJWTCWG5QRQQFB363DFVFZDA, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.019248864 (524h17m26.481341347s) [noise=true , address=145.79.227.17:4969 , piece_id=R5HIP33QW5HLICARFWDUOW7I75YTDHZTJCIML5X52VKHLPL3J4YQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.075991585 (524h17m26.538084068s) [noise=true , address=145.79.227.4:4187 , piece_id=TQVPL6ZA6LHKAJVL6XWSYASZD3MCKKV3553CRFD72XWTVWY62STQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.118655146 (524h17m26.580747619s) [noise=true , address=145.79.227.14:4838 , piece_id=6K6A4CNEHGB3X6DEBLF5KHLUT4GE6BQBW6IE3RHBML2NWPN255AA, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.285934795 (524h17m26.748027278s) [noise=true , address=77.90.31.11:4555 , piece_id=U5RYZSD643KSUI4XLL26EAN6FQRPMG62GQMMCQRPEEG2XKSNBIZQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.744525465 (524h17m27.206617948s) [noise=true , address=45.91.116.9:4262 , piece_id=X53JZGRLZLX53UZLLCXXUQCFSQD2FO76X74OM45G26HNDU55UXXQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:42.218961989 (524h17m27.681054462s) [noise=true , address=45.91.116.11:4117 , piece_id=VH7ZJKSZ2NUXQJZE2IKNWP5PAL3TN2NNHD3ATZUQHGKKYMSSM3LQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:42.313638675 (524h17m27.775731158s) [noise=true , address=45.91.116.14:4189 , piece_id=UMQRQYPZ7SWUCO2N7JDCEVI5U5P36FDL3SUM6HIYJVWVOZTWNXQQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:42.434283559 (524h17m27.896376053s) [noise=true , address=45.91.116.8:4045 , piece_id=EDXBNLKKRZMT6MKK2KOIZM7DDKA3IHB6UMCXX4YKZF7JY2T3S5CQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:42.434321832 (524h17m27.896414305s) [seg_pos={0 0}, upload=461218]: Segment reached optimal threshold of 13 pieces. Oct 15 09:48:42.434605041 (524h17m27.896697534s) [noise=true , address=45.91.116.5:4299 , piece_id=4ARMXXE5LBRE2OZQEIITELKPTW4TISZWTD6XZGI4UJV7UCLUXAEQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX): protocol: expected piece hash; context canceled; EOF Oct 15 09:48:42.434666138 (524h17m27.896758621s) [noise=true , address=45.91.116.5:4299 , piece_id=4ARMXXE5LBRE2OZQEIITELKPTW4TISZWTD6XZGI4UJV7UCLUXAEQ, seg_pos={0 0}, upload=461218]: Long tail cancellation due to optimal threshold Oct 15 09:48:42.434692849 (524h17m27.896785432s) [noise=true , address=45.91.116.12:4226 , piece_id=DRXRGO7OXEKVS5ACL6FC7RKLGBBD6KNI7X6E5DM5D5VV5PEKN7FQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1): protocol: expected piece hash; context canceled; EOF Oct 15 09:48:42.434706184 (524h17m27.896798657s) [noise=true , address=45.91.116.12:4226 , piece_id=DRXRGO7OXEKVS5ACL6FC7RKLGBBD6KNI7X6E5DM5D5VV5PEKN7FQ, seg_pos={0 0}, upload=461218]: Long tail cancellation due to optimal threshold Oct 15 09:48:42.434718318 (524h17m27.896810911s) [noise=true , address=45.91.116.4:4940 , piece_id=3MCSUDYN73NIVJTGKPOVY6N3MALB3GBXI7I5NAX4HXPDHVMQ52BA, seg_pos={0 0}, upload=461218]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe): protocol: expected piece hash; context canceled; EOF Oct 15 09:48:42.434728447 (524h17m27.89682093s) [noise=true , address=45.91.116.4:4940 , piece_id=3MCSUDYN73NIVJTGKPOVY6N3MALB3GBXI7I5NAX4HXPDHVMQ52BA, seg_pos={0 0}, upload=461218]: Long tail cancellation due to optimal threshold Oct 15 09:48:42.434760458 (524h17m27.896852931s) [seg_pos={0 0}, upload=461218]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 15 09:48:42.434866900 (524h17m27.896959384s) [upload=461218]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 15 09:48:42.434873884 (524h17m27.896966367s) [upload=461218]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 15 09:48:42.760633614 (524h17m28.222726087s) [upload=461218]: Done uploading segments... Oct 15 09:48:42.760655435 (524h17m28.222747918s) [upload=461218]: Upload finished. err: <nil> Duration: 6.392919892s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..X...X.....................................................] duration: 6.202304229s FAIL (2025-10-15 10:29:09 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, Piece ID: VH7ZJKSZ2NUXQJZE2IKNWP5PAL3TN2NNHD3ATZUQHGKKYMSSM3LQ) context deadline exceeded; piecestore: (Node ID: 1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N, Piece ID: UMQRQYPZ7SWUCO2N7JDCEVI5U5P36FDL3SUM6HIYJVWVOZTWNXQQ) context deadline exceeded ======LOGS====== Oct 15 09:48:38.801058027 (524h17m24.26315051s) [upload=461218]: Starting upload Oct 15 09:48:38.801138119 (524h17m24.263230592s) [upload=461218]: Done starting upload Oct 15 09:48:38.801288545 (524h17m24.263381028s) [upload=461218]: Uploading segments... Oct 15 09:48:38.801442509 (524h17m24.263535002s) [upload=461218]: Got next segment. Inline: false Oct 15 09:48:38.801486663 (524h17m24.263579136s) [upload=461218]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 15 09:48:38.801493245 (524h17m24.263585728s) [upload=461218]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 15 09:48:39.066131918 (524h17m24.528224401s) [seg_pos={0 0}, upload=461218]: Begin upload segment... Oct 15 09:48:39.066205549 (524h17m24.528298022s) [seg_pos={0 0}, upload=461218]: Done begin upload segment. Oct 15 09:48:39.066243090 (524h17m24.528335573s) [noise=true , address=45.91.116.5:4299 , piece_id=4ARMXXE5LBRE2OZQEIITELKPTW4TISZWTD6XZGI4UJV7UCLUXAEQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.066254431 (524h17m24.528346915s) [noise=true , address=45.91.116.11:4117 , piece_id=VH7ZJKSZ2NUXQJZE2IKNWP5PAL3TN2NNHD3ATZUQHGKKYMSSM3LQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.066874873 (524h17m24.528967356s) [noise=true , address=145.79.227.7:4294 , piece_id=H4ZNVIXORNO4MTXOVYDQZI4IJE4VXHZWC6JI35YKOR3UESSXJNTQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.067301676 (524h17m24.529394159s) [noise=true , address=45.91.116.9:4262 , piece_id=X53JZGRLZLX53UZLLCXXUQCFSQD2FO76X74OM45G26HNDU55UXXQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.067705735 (524h17m24.529798218s) [noise=true , address=145.79.227.4:4187 , piece_id=TQVPL6ZA6LHKAJVL6XWSYASZD3MCKKV3553CRFD72XWTVWY62STQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.069183369 (524h17m24.531275862s) [noise=true , address=145.79.227.5:4222 , piece_id=C2JSPN2QVSGSXG6ZW3ZMOFX5YFNN65XY7E62N5OEHOOMK3NSO4KQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.070534782 (524h17m24.532627265s) [noise=true , address=77.90.31.11:4555 , piece_id=U5RYZSD643KSUI4XLL26EAN6FQRPMG62GQMMCQRPEEG2XKSNBIZQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.072356982 (524h17m24.534449466s) [noise=true , address=145.79.227.11:4730 , piece_id=VO2DSM5LT43QAFPWOY3W3PVEJ44GZJWTCWG5QRQQFB363DFVFZDA, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.073961167 (524h17m24.53605364s) [noise=true , address=45.91.116.4:4940 , piece_id=3MCSUDYN73NIVJTGKPOVY6N3MALB3GBXI7I5NAX4HXPDHVMQ52BA, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.075716960 (524h17m24.537809433s) [noise=true , address=145.79.227.16:4910 , piece_id=BIHEJDH7RPI4UCPUNV453QHSEVCLA54PSNNQB2II37XZDRXFZNWQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.079266419 (524h17m24.541358902s) [noise=true , address=145.79.227.14:4838 , piece_id=6K6A4CNEHGB3X6DEBLF5KHLUT4GE6BQBW6IE3RHBML2NWPN255AA, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.080943032 (524h17m24.543035505s) [noise=true , address=45.91.116.12:4226 , piece_id=DRXRGO7OXEKVS5ACL6FC7RKLGBBD6KNI7X6E5DM5D5VV5PEKN7FQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.081340479 (524h17m24.543432962s) [noise=true , address=45.91.116.8:4045 , piece_id=EDXBNLKKRZMT6MKK2KOIZM7DDKA3IHB6UMCXX4YKZF7JY2T3S5CQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.083616223 (524h17m24.545708706s) [noise=true , address=145.79.227.14:4840 , piece_id=B4ZVPRMVH7Y3TCWCLPO7DWRVMJL7VSI73CIT4PEHCMTIEURQKB2Q, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.083633946 (524h17m24.545726429s) [noise=true , address=45.91.116.14:4189 , piece_id=UMQRQYPZ7SWUCO2N7JDCEVI5U5P36FDL3SUM6HIYJVWVOZTWNXQQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.085493247 (524h17m24.54758573s) [noise=true , address=145.79.227.17:4969 , piece_id=R5HIP33QW5HLICARFWDUOW7I75YTDHZTJCIML5X52VKHLPL3J4YQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:40.727263388 (524h17m26.189355871s) [upload=461218]: Next returned nil segment Oct 15 09:48:40.727310076 (524h17m26.189402549s) [upload=461218]: Waiting for error group managing segments... Oct 15 09:48:41.004514306 (524h17m26.466606789s) [noise=true , address=145.79.227.7:4294 , piece_id=H4ZNVIXORNO4MTXOVYDQZI4IJE4VXHZWC6JI35YKOR3UESSXJNTQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.006116557 (524h17m26.46820904s) [noise=true , address=145.79.227.5:4222 , piece_id=C2JSPN2QVSGSXG6ZW3ZMOFX5YFNN65XY7E62N5OEHOOMK3NSO4KQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.012963184 (524h17m26.475055667s) [noise=true , address=145.79.227.16:4910 , piece_id=BIHEJDH7RPI4UCPUNV453QHSEVCLA54PSNNQB2II37XZDRXFZNWQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.012994684 (524h17m26.475087167s) [noise=true , address=145.79.227.14:4840 , piece_id=B4ZVPRMVH7Y3TCWCLPO7DWRVMJL7VSI73CIT4PEHCMTIEURQKB2Q, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.016649956 (524h17m26.478742439s) [noise=true , address=145.79.227.11:4730 , piece_id=VO2DSM5LT43QAFPWOY3W3PVEJ44GZJWTCWG5QRQQFB363DFVFZDA, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.019248864 (524h17m26.481341347s) [noise=true , address=145.79.227.17:4969 , piece_id=R5HIP33QW5HLICARFWDUOW7I75YTDHZTJCIML5X52VKHLPL3J4YQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.075991585 (524h17m26.538084068s) [noise=true , address=145.79.227.4:4187 , piece_id=TQVPL6ZA6LHKAJVL6XWSYASZD3MCKKV3553CRFD72XWTVWY62STQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.118655146 (524h17m26.580747619s) [noise=true , address=145.79.227.14:4838 , piece_id=6K6A4CNEHGB3X6DEBLF5KHLUT4GE6BQBW6IE3RHBML2NWPN255AA, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.285934795 (524h17m26.748027278s) [noise=true , address=77.90.31.11:4555 , piece_id=U5RYZSD643KSUI4XLL26EAN6FQRPMG62GQMMCQRPEEG2XKSNBIZQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.744525465 (524h17m27.206617948s) [noise=true , address=45.91.116.9:4262 , piece_id=X53JZGRLZLX53UZLLCXXUQCFSQD2FO76X74OM45G26HNDU55UXXQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:42.218961989 (524h17m27.681054462s) [noise=true , address=45.91.116.11:4117 , piece_id=VH7ZJKSZ2NUXQJZE2IKNWP5PAL3TN2NNHD3ATZUQHGKKYMSSM3LQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:42.313638675 (524h17m27.775731158s) [noise=true , address=45.91.116.14:4189 , piece_id=UMQRQYPZ7SWUCO2N7JDCEVI5U5P36FDL3SUM6HIYJVWVOZTWNXQQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:42.434283559 (524h17m27.896376053s) [noise=true , address=45.91.116.8:4045 , piece_id=EDXBNLKKRZMT6MKK2KOIZM7DDKA3IHB6UMCXX4YKZF7JY2T3S5CQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:42.434321832 (524h17m27.896414305s) [seg_pos={0 0}, upload=461218]: Segment reached optimal threshold of 13 pieces. Oct 15 09:48:42.434605041 (524h17m27.896697534s) [noise=true , address=45.91.116.5:4299 , piece_id=4ARMXXE5LBRE2OZQEIITELKPTW4TISZWTD6XZGI4UJV7UCLUXAEQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX): protocol: expected piece hash; context canceled; EOF Oct 15 09:48:42.434666138 (524h17m27.896758621s) [noise=true , address=45.91.116.5:4299 , piece_id=4ARMXXE5LBRE2OZQEIITELKPTW4TISZWTD6XZGI4UJV7UCLUXAEQ, seg_pos={0 0}, upload=461218]: Long tail cancellation due to optimal threshold Oct 15 09:48:42.434692849 (524h17m27.896785432s) [noise=true , address=45.91.116.12:4226 , piece_id=DRXRGO7OXEKVS5ACL6FC7RKLGBBD6KNI7X6E5DM5D5VV5PEKN7FQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1): protocol: expected piece hash; context canceled; EOF Oct 15 09:48:42.434706184 (524h17m27.896798657s) [noise=true , address=45.91.116.12:4226 , piece_id=DRXRGO7OXEKVS5ACL6FC7RKLGBBD6KNI7X6E5DM5D5VV5PEKN7FQ, seg_pos={0 0}, upload=461218]: Long tail cancellation due to optimal threshold Oct 15 09:48:42.434718318 (524h17m27.896810911s) [noise=true , address=45.91.116.4:4940 , piece_id=3MCSUDYN73NIVJTGKPOVY6N3MALB3GBXI7I5NAX4HXPDHVMQ52BA, seg_pos={0 0}, upload=461218]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe): protocol: expected piece hash; context canceled; EOF Oct 15 09:48:42.434728447 (524h17m27.89682093s) [noise=true , address=45.91.116.4:4940 , piece_id=3MCSUDYN73NIVJTGKPOVY6N3MALB3GBXI7I5NAX4HXPDHVMQ52BA, seg_pos={0 0}, upload=461218]: Long tail cancellation due to optimal threshold Oct 15 09:48:42.434760458 (524h17m27.896852931s) [seg_pos={0 0}, upload=461218]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 15 09:48:42.434866900 (524h17m27.896959384s) [upload=461218]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 15 09:48:42.434873884 (524h17m27.896966367s) [upload=461218]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 15 09:48:42.760633614 (524h17m28.222726087s) [upload=461218]: Done uploading segments... Oct 15 09:48:42.760655435 (524h17m28.222747918s) [upload=461218]: Upload finished. err: <nil> Duration: 6.202341329s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...X...X....................................................] duration: 6.081043894s FAIL (2025-10-15 10:28:16 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, Piece ID: VH7ZJKSZ2NUXQJZE2IKNWP5PAL3TN2NNHD3ATZUQHGKKYMSSM3LQ) context deadline exceeded; piecestore: (Node ID: 1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N, Piece ID: UMQRQYPZ7SWUCO2N7JDCEVI5U5P36FDL3SUM6HIYJVWVOZTWNXQQ) context deadline exceeded ======LOGS====== Oct 15 09:48:38.801058027 (524h17m24.26315051s) [upload=461218]: Starting upload Oct 15 09:48:38.801138119 (524h17m24.263230592s) [upload=461218]: Done starting upload Oct 15 09:48:38.801288545 (524h17m24.263381028s) [upload=461218]: Uploading segments... Oct 15 09:48:38.801442509 (524h17m24.263535002s) [upload=461218]: Got next segment. Inline: false Oct 15 09:48:38.801486663 (524h17m24.263579136s) [upload=461218]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 15 09:48:38.801493245 (524h17m24.263585728s) [upload=461218]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 15 09:48:39.066131918 (524h17m24.528224401s) [seg_pos={0 0}, upload=461218]: Begin upload segment... Oct 15 09:48:39.066205549 (524h17m24.528298022s) [seg_pos={0 0}, upload=461218]: Done begin upload segment. Oct 15 09:48:39.066243090 (524h17m24.528335573s) [noise=true , address=45.91.116.5:4299 , piece_id=4ARMXXE5LBRE2OZQEIITELKPTW4TISZWTD6XZGI4UJV7UCLUXAEQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.066254431 (524h17m24.528346915s) [noise=true , address=45.91.116.11:4117 , piece_id=VH7ZJKSZ2NUXQJZE2IKNWP5PAL3TN2NNHD3ATZUQHGKKYMSSM3LQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.066874873 (524h17m24.528967356s) [noise=true , address=145.79.227.7:4294 , piece_id=H4ZNVIXORNO4MTXOVYDQZI4IJE4VXHZWC6JI35YKOR3UESSXJNTQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.067301676 (524h17m24.529394159s) [noise=true , address=45.91.116.9:4262 , piece_id=X53JZGRLZLX53UZLLCXXUQCFSQD2FO76X74OM45G26HNDU55UXXQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.067705735 (524h17m24.529798218s) [noise=true , address=145.79.227.4:4187 , piece_id=TQVPL6ZA6LHKAJVL6XWSYASZD3MCKKV3553CRFD72XWTVWY62STQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.069183369 (524h17m24.531275862s) [noise=true , address=145.79.227.5:4222 , piece_id=C2JSPN2QVSGSXG6ZW3ZMOFX5YFNN65XY7E62N5OEHOOMK3NSO4KQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.070534782 (524h17m24.532627265s) [noise=true , address=77.90.31.11:4555 , piece_id=U5RYZSD643KSUI4XLL26EAN6FQRPMG62GQMMCQRPEEG2XKSNBIZQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.072356982 (524h17m24.534449466s) [noise=true , address=145.79.227.11:4730 , piece_id=VO2DSM5LT43QAFPWOY3W3PVEJ44GZJWTCWG5QRQQFB363DFVFZDA, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.073961167 (524h17m24.53605364s) [noise=true , address=45.91.116.4:4940 , piece_id=3MCSUDYN73NIVJTGKPOVY6N3MALB3GBXI7I5NAX4HXPDHVMQ52BA, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.075716960 (524h17m24.537809433s) [noise=true , address=145.79.227.16:4910 , piece_id=BIHEJDH7RPI4UCPUNV453QHSEVCLA54PSNNQB2II37XZDRXFZNWQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.079266419 (524h17m24.541358902s) [noise=true , address=145.79.227.14:4838 , piece_id=6K6A4CNEHGB3X6DEBLF5KHLUT4GE6BQBW6IE3RHBML2NWPN255AA, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.080943032 (524h17m24.543035505s) [noise=true , address=45.91.116.12:4226 , piece_id=DRXRGO7OXEKVS5ACL6FC7RKLGBBD6KNI7X6E5DM5D5VV5PEKN7FQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.081340479 (524h17m24.543432962s) [noise=true , address=45.91.116.8:4045 , piece_id=EDXBNLKKRZMT6MKK2KOIZM7DDKA3IHB6UMCXX4YKZF7JY2T3S5CQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.083616223 (524h17m24.545708706s) [noise=true , address=145.79.227.14:4840 , piece_id=B4ZVPRMVH7Y3TCWCLPO7DWRVMJL7VSI73CIT4PEHCMTIEURQKB2Q, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.083633946 (524h17m24.545726429s) [noise=true , address=45.91.116.14:4189 , piece_id=UMQRQYPZ7SWUCO2N7JDCEVI5U5P36FDL3SUM6HIYJVWVOZTWNXQQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.085493247 (524h17m24.54758573s) [noise=true , address=145.79.227.17:4969 , piece_id=R5HIP33QW5HLICARFWDUOW7I75YTDHZTJCIML5X52VKHLPL3J4YQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:40.727263388 (524h17m26.189355871s) [upload=461218]: Next returned nil segment Oct 15 09:48:40.727310076 (524h17m26.189402549s) [upload=461218]: Waiting for error group managing segments... Oct 15 09:48:41.004514306 (524h17m26.466606789s) [noise=true , address=145.79.227.7:4294 , piece_id=H4ZNVIXORNO4MTXOVYDQZI4IJE4VXHZWC6JI35YKOR3UESSXJNTQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.006116557 (524h17m26.46820904s) [noise=true , address=145.79.227.5:4222 , piece_id=C2JSPN2QVSGSXG6ZW3ZMOFX5YFNN65XY7E62N5OEHOOMK3NSO4KQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.012963184 (524h17m26.475055667s) [noise=true , address=145.79.227.16:4910 , piece_id=BIHEJDH7RPI4UCPUNV453QHSEVCLA54PSNNQB2II37XZDRXFZNWQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.012994684 (524h17m26.475087167s) [noise=true , address=145.79.227.14:4840 , piece_id=B4ZVPRMVH7Y3TCWCLPO7DWRVMJL7VSI73CIT4PEHCMTIEURQKB2Q, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.016649956 (524h17m26.478742439s) [noise=true , address=145.79.227.11:4730 , piece_id=VO2DSM5LT43QAFPWOY3W3PVEJ44GZJWTCWG5QRQQFB363DFVFZDA, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.019248864 (524h17m26.481341347s) [noise=true , address=145.79.227.17:4969 , piece_id=R5HIP33QW5HLICARFWDUOW7I75YTDHZTJCIML5X52VKHLPL3J4YQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.075991585 (524h17m26.538084068s) [noise=true , address=145.79.227.4:4187 , piece_id=TQVPL6ZA6LHKAJVL6XWSYASZD3MCKKV3553CRFD72XWTVWY62STQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.118655146 (524h17m26.580747619s) [noise=true , address=145.79.227.14:4838 , piece_id=6K6A4CNEHGB3X6DEBLF5KHLUT4GE6BQBW6IE3RHBML2NWPN255AA, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.285934795 (524h17m26.748027278s) [noise=true , address=77.90.31.11:4555 , piece_id=U5RYZSD643KSUI4XLL26EAN6FQRPMG62GQMMCQRPEEG2XKSNBIZQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.744525465 (524h17m27.206617948s) [noise=true , address=45.91.116.9:4262 , piece_id=X53JZGRLZLX53UZLLCXXUQCFSQD2FO76X74OM45G26HNDU55UXXQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:42.218961989 (524h17m27.681054462s) [noise=true , address=45.91.116.11:4117 , piece_id=VH7ZJKSZ2NUXQJZE2IKNWP5PAL3TN2NNHD3ATZUQHGKKYMSSM3LQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:42.313638675 (524h17m27.775731158s) [noise=true , address=45.91.116.14:4189 , piece_id=UMQRQYPZ7SWUCO2N7JDCEVI5U5P36FDL3SUM6HIYJVWVOZTWNXQQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:42.434283559 (524h17m27.896376053s) [noise=true , address=45.91.116.8:4045 , piece_id=EDXBNLKKRZMT6MKK2KOIZM7DDKA3IHB6UMCXX4YKZF7JY2T3S5CQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:42.434321832 (524h17m27.896414305s) [seg_pos={0 0}, upload=461218]: Segment reached optimal threshold of 13 pieces. Oct 15 09:48:42.434605041 (524h17m27.896697534s) [noise=true , address=45.91.116.5:4299 , piece_id=4ARMXXE5LBRE2OZQEIITELKPTW4TISZWTD6XZGI4UJV7UCLUXAEQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX): protocol: expected piece hash; context canceled; EOF Oct 15 09:48:42.434666138 (524h17m27.896758621s) [noise=true , address=45.91.116.5:4299 , piece_id=4ARMXXE5LBRE2OZQEIITELKPTW4TISZWTD6XZGI4UJV7UCLUXAEQ, seg_pos={0 0}, upload=461218]: Long tail cancellation due to optimal threshold Oct 15 09:48:42.434692849 (524h17m27.896785432s) [noise=true , address=45.91.116.12:4226 , piece_id=DRXRGO7OXEKVS5ACL6FC7RKLGBBD6KNI7X6E5DM5D5VV5PEKN7FQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1): protocol: expected piece hash; context canceled; EOF Oct 15 09:48:42.434706184 (524h17m27.896798657s) [noise=true , address=45.91.116.12:4226 , piece_id=DRXRGO7OXEKVS5ACL6FC7RKLGBBD6KNI7X6E5DM5D5VV5PEKN7FQ, seg_pos={0 0}, upload=461218]: Long tail cancellation due to optimal threshold Oct 15 09:48:42.434718318 (524h17m27.896810911s) [noise=true , address=45.91.116.4:4940 , piece_id=3MCSUDYN73NIVJTGKPOVY6N3MALB3GBXI7I5NAX4HXPDHVMQ52BA, seg_pos={0 0}, upload=461218]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe): protocol: expected piece hash; context canceled; EOF Oct 15 09:48:42.434728447 (524h17m27.89682093s) [noise=true , address=45.91.116.4:4940 , piece_id=3MCSUDYN73NIVJTGKPOVY6N3MALB3GBXI7I5NAX4HXPDHVMQ52BA, seg_pos={0 0}, upload=461218]: Long tail cancellation due to optimal threshold Oct 15 09:48:42.434760458 (524h17m27.896852931s) [seg_pos={0 0}, upload=461218]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 15 09:48:42.434866900 (524h17m27.896959384s) [upload=461218]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 15 09:48:42.434873884 (524h17m27.896966367s) [upload=461218]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 15 09:48:42.760633614 (524h17m28.222726087s) [upload=461218]: Done uploading segments... Oct 15 09:48:42.760655435 (524h17m28.222747918s) [upload=461218]: Upload finished. err: <nil> Duration: 6.081080003s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [....X...X...................................................] duration: 6.028003516s FAIL (2025-10-15 10:27:26 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, Piece ID: VH7ZJKSZ2NUXQJZE2IKNWP5PAL3TN2NNHD3ATZUQHGKKYMSSM3LQ) context deadline exceeded; piecestore: (Node ID: 1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N, Piece ID: UMQRQYPZ7SWUCO2N7JDCEVI5U5P36FDL3SUM6HIYJVWVOZTWNXQQ) context deadline exceeded ======LOGS====== Oct 15 09:48:38.801058027 (524h17m24.26315051s) [upload=461218]: Starting upload Oct 15 09:48:38.801138119 (524h17m24.263230592s) [upload=461218]: Done starting upload Oct 15 09:48:38.801288545 (524h17m24.263381028s) [upload=461218]: Uploading segments... Oct 15 09:48:38.801442509 (524h17m24.263535002s) [upload=461218]: Got next segment. Inline: false Oct 15 09:48:38.801486663 (524h17m24.263579136s) [upload=461218]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 15 09:48:38.801493245 (524h17m24.263585728s) [upload=461218]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 15 09:48:39.066131918 (524h17m24.528224401s) [seg_pos={0 0}, upload=461218]: Begin upload segment... Oct 15 09:48:39.066205549 (524h17m24.528298022s) [seg_pos={0 0}, upload=461218]: Done begin upload segment. Oct 15 09:48:39.066243090 (524h17m24.528335573s) [noise=true , address=45.91.116.5:4299 , piece_id=4ARMXXE5LBRE2OZQEIITELKPTW4TISZWTD6XZGI4UJV7UCLUXAEQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.066254431 (524h17m24.528346915s) [noise=true , address=45.91.116.11:4117 , piece_id=VH7ZJKSZ2NUXQJZE2IKNWP5PAL3TN2NNHD3ATZUQHGKKYMSSM3LQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.066874873 (524h17m24.528967356s) [noise=true , address=145.79.227.7:4294 , piece_id=H4ZNVIXORNO4MTXOVYDQZI4IJE4VXHZWC6JI35YKOR3UESSXJNTQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.067301676 (524h17m24.529394159s) [noise=true , address=45.91.116.9:4262 , piece_id=X53JZGRLZLX53UZLLCXXUQCFSQD2FO76X74OM45G26HNDU55UXXQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.067705735 (524h17m24.529798218s) [noise=true , address=145.79.227.4:4187 , piece_id=TQVPL6ZA6LHKAJVL6XWSYASZD3MCKKV3553CRFD72XWTVWY62STQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.069183369 (524h17m24.531275862s) [noise=true , address=145.79.227.5:4222 , piece_id=C2JSPN2QVSGSXG6ZW3ZMOFX5YFNN65XY7E62N5OEHOOMK3NSO4KQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.070534782 (524h17m24.532627265s) [noise=true , address=77.90.31.11:4555 , piece_id=U5RYZSD643KSUI4XLL26EAN6FQRPMG62GQMMCQRPEEG2XKSNBIZQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.072356982 (524h17m24.534449466s) [noise=true , address=145.79.227.11:4730 , piece_id=VO2DSM5LT43QAFPWOY3W3PVEJ44GZJWTCWG5QRQQFB363DFVFZDA, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.073961167 (524h17m24.53605364s) [noise=true , address=45.91.116.4:4940 , piece_id=3MCSUDYN73NIVJTGKPOVY6N3MALB3GBXI7I5NAX4HXPDHVMQ52BA, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.075716960 (524h17m24.537809433s) [noise=true , address=145.79.227.16:4910 , piece_id=BIHEJDH7RPI4UCPUNV453QHSEVCLA54PSNNQB2II37XZDRXFZNWQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.079266419 (524h17m24.541358902s) [noise=true , address=145.79.227.14:4838 , piece_id=6K6A4CNEHGB3X6DEBLF5KHLUT4GE6BQBW6IE3RHBML2NWPN255AA, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.080943032 (524h17m24.543035505s) [noise=true , address=45.91.116.12:4226 , piece_id=DRXRGO7OXEKVS5ACL6FC7RKLGBBD6KNI7X6E5DM5D5VV5PEKN7FQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.081340479 (524h17m24.543432962s) [noise=true , address=45.91.116.8:4045 , piece_id=EDXBNLKKRZMT6MKK2KOIZM7DDKA3IHB6UMCXX4YKZF7JY2T3S5CQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.083616223 (524h17m24.545708706s) [noise=true , address=145.79.227.14:4840 , piece_id=B4ZVPRMVH7Y3TCWCLPO7DWRVMJL7VSI73CIT4PEHCMTIEURQKB2Q, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.083633946 (524h17m24.545726429s) [noise=true , address=45.91.116.14:4189 , piece_id=UMQRQYPZ7SWUCO2N7JDCEVI5U5P36FDL3SUM6HIYJVWVOZTWNXQQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.085493247 (524h17m24.54758573s) [noise=true , address=145.79.227.17:4969 , piece_id=R5HIP33QW5HLICARFWDUOW7I75YTDHZTJCIML5X52VKHLPL3J4YQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:40.727263388 (524h17m26.189355871s) [upload=461218]: Next returned nil segment Oct 15 09:48:40.727310076 (524h17m26.189402549s) [upload=461218]: Waiting for error group managing segments... Oct 15 09:48:41.004514306 (524h17m26.466606789s) [noise=true , address=145.79.227.7:4294 , piece_id=H4ZNVIXORNO4MTXOVYDQZI4IJE4VXHZWC6JI35YKOR3UESSXJNTQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.006116557 (524h17m26.46820904s) [noise=true , address=145.79.227.5:4222 , piece_id=C2JSPN2QVSGSXG6ZW3ZMOFX5YFNN65XY7E62N5OEHOOMK3NSO4KQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.012963184 (524h17m26.475055667s) [noise=true , address=145.79.227.16:4910 , piece_id=BIHEJDH7RPI4UCPUNV453QHSEVCLA54PSNNQB2II37XZDRXFZNWQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.012994684 (524h17m26.475087167s) [noise=true , address=145.79.227.14:4840 , piece_id=B4ZVPRMVH7Y3TCWCLPO7DWRVMJL7VSI73CIT4PEHCMTIEURQKB2Q, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.016649956 (524h17m26.478742439s) [noise=true , address=145.79.227.11:4730 , piece_id=VO2DSM5LT43QAFPWOY3W3PVEJ44GZJWTCWG5QRQQFB363DFVFZDA, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.019248864 (524h17m26.481341347s) [noise=true , address=145.79.227.17:4969 , piece_id=R5HIP33QW5HLICARFWDUOW7I75YTDHZTJCIML5X52VKHLPL3J4YQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.075991585 (524h17m26.538084068s) [noise=true , address=145.79.227.4:4187 , piece_id=TQVPL6ZA6LHKAJVL6XWSYASZD3MCKKV3553CRFD72XWTVWY62STQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.118655146 (524h17m26.580747619s) [noise=true , address=145.79.227.14:4838 , piece_id=6K6A4CNEHGB3X6DEBLF5KHLUT4GE6BQBW6IE3RHBML2NWPN255AA, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.285934795 (524h17m26.748027278s) [noise=true , address=77.90.31.11:4555 , piece_id=U5RYZSD643KSUI4XLL26EAN6FQRPMG62GQMMCQRPEEG2XKSNBIZQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.744525465 (524h17m27.206617948s) [noise=true , address=45.91.116.9:4262 , piece_id=X53JZGRLZLX53UZLLCXXUQCFSQD2FO76X74OM45G26HNDU55UXXQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:42.218961989 (524h17m27.681054462s) [noise=true , address=45.91.116.11:4117 , piece_id=VH7ZJKSZ2NUXQJZE2IKNWP5PAL3TN2NNHD3ATZUQHGKKYMSSM3LQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:42.313638675 (524h17m27.775731158s) [noise=true , address=45.91.116.14:4189 , piece_id=UMQRQYPZ7SWUCO2N7JDCEVI5U5P36FDL3SUM6HIYJVWVOZTWNXQQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:42.434283559 (524h17m27.896376053s) [noise=true , address=45.91.116.8:4045 , piece_id=EDXBNLKKRZMT6MKK2KOIZM7DDKA3IHB6UMCXX4YKZF7JY2T3S5CQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:42.434321832 (524h17m27.896414305s) [seg_pos={0 0}, upload=461218]: Segment reached optimal threshold of 13 pieces. Oct 15 09:48:42.434605041 (524h17m27.896697534s) [noise=true , address=45.91.116.5:4299 , piece_id=4ARMXXE5LBRE2OZQEIITELKPTW4TISZWTD6XZGI4UJV7UCLUXAEQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX): protocol: expected piece hash; context canceled; EOF Oct 15 09:48:42.434666138 (524h17m27.896758621s) [noise=true , address=45.91.116.5:4299 , piece_id=4ARMXXE5LBRE2OZQEIITELKPTW4TISZWTD6XZGI4UJV7UCLUXAEQ, seg_pos={0 0}, upload=461218]: Long tail cancellation due to optimal threshold Oct 15 09:48:42.434692849 (524h17m27.896785432s) [noise=true , address=45.91.116.12:4226 , piece_id=DRXRGO7OXEKVS5ACL6FC7RKLGBBD6KNI7X6E5DM5D5VV5PEKN7FQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1): protocol: expected piece hash; context canceled; EOF Oct 15 09:48:42.434706184 (524h17m27.896798657s) [noise=true , address=45.91.116.12:4226 , piece_id=DRXRGO7OXEKVS5ACL6FC7RKLGBBD6KNI7X6E5DM5D5VV5PEKN7FQ, seg_pos={0 0}, upload=461218]: Long tail cancellation due to optimal threshold Oct 15 09:48:42.434718318 (524h17m27.896810911s) [noise=true , address=45.91.116.4:4940 , piece_id=3MCSUDYN73NIVJTGKPOVY6N3MALB3GBXI7I5NAX4HXPDHVMQ52BA, seg_pos={0 0}, upload=461218]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe): protocol: expected piece hash; context canceled; EOF Oct 15 09:48:42.434728447 (524h17m27.89682093s) [noise=true , address=45.91.116.4:4940 , piece_id=3MCSUDYN73NIVJTGKPOVY6N3MALB3GBXI7I5NAX4HXPDHVMQ52BA, seg_pos={0 0}, upload=461218]: Long tail cancellation due to optimal threshold Oct 15 09:48:42.434760458 (524h17m27.896852931s) [seg_pos={0 0}, upload=461218]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 15 09:48:42.434866900 (524h17m27.896959384s) [upload=461218]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 15 09:48:42.434873884 (524h17m27.896966367s) [upload=461218]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 15 09:48:42.760633614 (524h17m28.222726087s) [upload=461218]: Done uploading segments... Oct 15 09:48:42.760655435 (524h17m28.222747918s) [upload=461218]: Upload finished. err: <nil> Duration: 6.028032211s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.....X...X..................................................] duration: 5.947989827s FAIL (2025-10-15 10:26:44 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, Piece ID: VH7ZJKSZ2NUXQJZE2IKNWP5PAL3TN2NNHD3ATZUQHGKKYMSSM3LQ) context deadline exceeded; piecestore: (Node ID: 1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N, Piece ID: UMQRQYPZ7SWUCO2N7JDCEVI5U5P36FDL3SUM6HIYJVWVOZTWNXQQ) context deadline exceeded ======LOGS====== Oct 15 09:48:38.801058027 (524h17m24.26315051s) [upload=461218]: Starting upload Oct 15 09:48:38.801138119 (524h17m24.263230592s) [upload=461218]: Done starting upload Oct 15 09:48:38.801288545 (524h17m24.263381028s) [upload=461218]: Uploading segments... Oct 15 09:48:38.801442509 (524h17m24.263535002s) [upload=461218]: Got next segment. Inline: false Oct 15 09:48:38.801486663 (524h17m24.263579136s) [upload=461218]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 15 09:48:38.801493245 (524h17m24.263585728s) [upload=461218]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 15 09:48:39.066131918 (524h17m24.528224401s) [seg_pos={0 0}, upload=461218]: Begin upload segment... Oct 15 09:48:39.066205549 (524h17m24.528298022s) [seg_pos={0 0}, upload=461218]: Done begin upload segment. Oct 15 09:48:39.066243090 (524h17m24.528335573s) [noise=true , address=45.91.116.5:4299 , piece_id=4ARMXXE5LBRE2OZQEIITELKPTW4TISZWTD6XZGI4UJV7UCLUXAEQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.066254431 (524h17m24.528346915s) [noise=true , address=45.91.116.11:4117 , piece_id=VH7ZJKSZ2NUXQJZE2IKNWP5PAL3TN2NNHD3ATZUQHGKKYMSSM3LQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.066874873 (524h17m24.528967356s) [noise=true , address=145.79.227.7:4294 , piece_id=H4ZNVIXORNO4MTXOVYDQZI4IJE4VXHZWC6JI35YKOR3UESSXJNTQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.067301676 (524h17m24.529394159s) [noise=true , address=45.91.116.9:4262 , piece_id=X53JZGRLZLX53UZLLCXXUQCFSQD2FO76X74OM45G26HNDU55UXXQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.067705735 (524h17m24.529798218s) [noise=true , address=145.79.227.4:4187 , piece_id=TQVPL6ZA6LHKAJVL6XWSYASZD3MCKKV3553CRFD72XWTVWY62STQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.069183369 (524h17m24.531275862s) [noise=true , address=145.79.227.5:4222 , piece_id=C2JSPN2QVSGSXG6ZW3ZMOFX5YFNN65XY7E62N5OEHOOMK3NSO4KQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.070534782 (524h17m24.532627265s) [noise=true , address=77.90.31.11:4555 , piece_id=U5RYZSD643KSUI4XLL26EAN6FQRPMG62GQMMCQRPEEG2XKSNBIZQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.072356982 (524h17m24.534449466s) [noise=true , address=145.79.227.11:4730 , piece_id=VO2DSM5LT43QAFPWOY3W3PVEJ44GZJWTCWG5QRQQFB363DFVFZDA, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.073961167 (524h17m24.53605364s) [noise=true , address=45.91.116.4:4940 , piece_id=3MCSUDYN73NIVJTGKPOVY6N3MALB3GBXI7I5NAX4HXPDHVMQ52BA, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.075716960 (524h17m24.537809433s) [noise=true , address=145.79.227.16:4910 , piece_id=BIHEJDH7RPI4UCPUNV453QHSEVCLA54PSNNQB2II37XZDRXFZNWQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.079266419 (524h17m24.541358902s) [noise=true , address=145.79.227.14:4838 , piece_id=6K6A4CNEHGB3X6DEBLF5KHLUT4GE6BQBW6IE3RHBML2NWPN255AA, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.080943032 (524h17m24.543035505s) [noise=true , address=45.91.116.12:4226 , piece_id=DRXRGO7OXEKVS5ACL6FC7RKLGBBD6KNI7X6E5DM5D5VV5PEKN7FQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.081340479 (524h17m24.543432962s) [noise=true , address=45.91.116.8:4045 , piece_id=EDXBNLKKRZMT6MKK2KOIZM7DDKA3IHB6UMCXX4YKZF7JY2T3S5CQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.083616223 (524h17m24.545708706s) [noise=true , address=145.79.227.14:4840 , piece_id=B4ZVPRMVH7Y3TCWCLPO7DWRVMJL7VSI73CIT4PEHCMTIEURQKB2Q, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.083633946 (524h17m24.545726429s) [noise=true , address=45.91.116.14:4189 , piece_id=UMQRQYPZ7SWUCO2N7JDCEVI5U5P36FDL3SUM6HIYJVWVOZTWNXQQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.085493247 (524h17m24.54758573s) [noise=true , address=145.79.227.17:4969 , piece_id=R5HIP33QW5HLICARFWDUOW7I75YTDHZTJCIML5X52VKHLPL3J4YQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:40.727263388 (524h17m26.189355871s) [upload=461218]: Next returned nil segment Oct 15 09:48:40.727310076 (524h17m26.189402549s) [upload=461218]: Waiting for error group managing segments... Oct 15 09:48:41.004514306 (524h17m26.466606789s) [noise=true , address=145.79.227.7:4294 , piece_id=H4ZNVIXORNO4MTXOVYDQZI4IJE4VXHZWC6JI35YKOR3UESSXJNTQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.006116557 (524h17m26.46820904s) [noise=true , address=145.79.227.5:4222 , piece_id=C2JSPN2QVSGSXG6ZW3ZMOFX5YFNN65XY7E62N5OEHOOMK3NSO4KQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.012963184 (524h17m26.475055667s) [noise=true , address=145.79.227.16:4910 , piece_id=BIHEJDH7RPI4UCPUNV453QHSEVCLA54PSNNQB2II37XZDRXFZNWQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.012994684 (524h17m26.475087167s) [noise=true , address=145.79.227.14:4840 , piece_id=B4ZVPRMVH7Y3TCWCLPO7DWRVMJL7VSI73CIT4PEHCMTIEURQKB2Q, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.016649956 (524h17m26.478742439s) [noise=true , address=145.79.227.11:4730 , piece_id=VO2DSM5LT43QAFPWOY3W3PVEJ44GZJWTCWG5QRQQFB363DFVFZDA, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.019248864 (524h17m26.481341347s) [noise=true , address=145.79.227.17:4969 , piece_id=R5HIP33QW5HLICARFWDUOW7I75YTDHZTJCIML5X52VKHLPL3J4YQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.075991585 (524h17m26.538084068s) [noise=true , address=145.79.227.4:4187 , piece_id=TQVPL6ZA6LHKAJVL6XWSYASZD3MCKKV3553CRFD72XWTVWY62STQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.118655146 (524h17m26.580747619s) [noise=true , address=145.79.227.14:4838 , piece_id=6K6A4CNEHGB3X6DEBLF5KHLUT4GE6BQBW6IE3RHBML2NWPN255AA, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.285934795 (524h17m26.748027278s) [noise=true , address=77.90.31.11:4555 , piece_id=U5RYZSD643KSUI4XLL26EAN6FQRPMG62GQMMCQRPEEG2XKSNBIZQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.744525465 (524h17m27.206617948s) [noise=true , address=45.91.116.9:4262 , piece_id=X53JZGRLZLX53UZLLCXXUQCFSQD2FO76X74OM45G26HNDU55UXXQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:42.218961989 (524h17m27.681054462s) [noise=true , address=45.91.116.11:4117 , piece_id=VH7ZJKSZ2NUXQJZE2IKNWP5PAL3TN2NNHD3ATZUQHGKKYMSSM3LQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:42.313638675 (524h17m27.775731158s) [noise=true , address=45.91.116.14:4189 , piece_id=UMQRQYPZ7SWUCO2N7JDCEVI5U5P36FDL3SUM6HIYJVWVOZTWNXQQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:42.434283559 (524h17m27.896376053s) [noise=true , address=45.91.116.8:4045 , piece_id=EDXBNLKKRZMT6MKK2KOIZM7DDKA3IHB6UMCXX4YKZF7JY2T3S5CQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:42.434321832 (524h17m27.896414305s) [seg_pos={0 0}, upload=461218]: Segment reached optimal threshold of 13 pieces. Oct 15 09:48:42.434605041 (524h17m27.896697534s) [noise=true , address=45.91.116.5:4299 , piece_id=4ARMXXE5LBRE2OZQEIITELKPTW4TISZWTD6XZGI4UJV7UCLUXAEQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX): protocol: expected piece hash; context canceled; EOF Oct 15 09:48:42.434666138 (524h17m27.896758621s) [noise=true , address=45.91.116.5:4299 , piece_id=4ARMXXE5LBRE2OZQEIITELKPTW4TISZWTD6XZGI4UJV7UCLUXAEQ, seg_pos={0 0}, upload=461218]: Long tail cancellation due to optimal threshold Oct 15 09:48:42.434692849 (524h17m27.896785432s) [noise=true , address=45.91.116.12:4226 , piece_id=DRXRGO7OXEKVS5ACL6FC7RKLGBBD6KNI7X6E5DM5D5VV5PEKN7FQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1): protocol: expected piece hash; context canceled; EOF Oct 15 09:48:42.434706184 (524h17m27.896798657s) [noise=true , address=45.91.116.12:4226 , piece_id=DRXRGO7OXEKVS5ACL6FC7RKLGBBD6KNI7X6E5DM5D5VV5PEKN7FQ, seg_pos={0 0}, upload=461218]: Long tail cancellation due to optimal threshold Oct 15 09:48:42.434718318 (524h17m27.896810911s) [noise=true , address=45.91.116.4:4940 , piece_id=3MCSUDYN73NIVJTGKPOVY6N3MALB3GBXI7I5NAX4HXPDHVMQ52BA, seg_pos={0 0}, upload=461218]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe): protocol: expected piece hash; context canceled; EOF Oct 15 09:48:42.434728447 (524h17m27.89682093s) [noise=true , address=45.91.116.4:4940 , piece_id=3MCSUDYN73NIVJTGKPOVY6N3MALB3GBXI7I5NAX4HXPDHVMQ52BA, seg_pos={0 0}, upload=461218]: Long tail cancellation due to optimal threshold Oct 15 09:48:42.434760458 (524h17m27.896852931s) [seg_pos={0 0}, upload=461218]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 15 09:48:42.434866900 (524h17m27.896959384s) [upload=461218]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 15 09:48:42.434873884 (524h17m27.896966367s) [upload=461218]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 15 09:48:42.760633614 (524h17m28.222726087s) [upload=461218]: Done uploading segments... Oct 15 09:48:42.760655435 (524h17m28.222747918s) [upload=461218]: Upload finished. err: <nil> Duration: 5.948017148s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [......X...X.................................................] duration: 9.437292616s FAIL (2025-10-15 10:26:24 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, Piece ID: VH7ZJKSZ2NUXQJZE2IKNWP5PAL3TN2NNHD3ATZUQHGKKYMSSM3LQ) context deadline exceeded; piecestore: (Node ID: 1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N, Piece ID: UMQRQYPZ7SWUCO2N7JDCEVI5U5P36FDL3SUM6HIYJVWVOZTWNXQQ) context deadline exceeded ======LOGS====== Oct 15 09:48:38.801058027 (524h17m24.26315051s) [upload=461218]: Starting upload Oct 15 09:48:38.801138119 (524h17m24.263230592s) [upload=461218]: Done starting upload Oct 15 09:48:38.801288545 (524h17m24.263381028s) [upload=461218]: Uploading segments... Oct 15 09:48:38.801442509 (524h17m24.263535002s) [upload=461218]: Got next segment. Inline: false Oct 15 09:48:38.801486663 (524h17m24.263579136s) [upload=461218]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 15 09:48:38.801493245 (524h17m24.263585728s) [upload=461218]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 15 09:48:39.066131918 (524h17m24.528224401s) [seg_pos={0 0}, upload=461218]: Begin upload segment... Oct 15 09:48:39.066205549 (524h17m24.528298022s) [seg_pos={0 0}, upload=461218]: Done begin upload segment. Oct 15 09:48:39.066243090 (524h17m24.528335573s) [noise=true , address=45.91.116.5:4299 , piece_id=4ARMXXE5LBRE2OZQEIITELKPTW4TISZWTD6XZGI4UJV7UCLUXAEQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.066254431 (524h17m24.528346915s) [noise=true , address=45.91.116.11:4117 , piece_id=VH7ZJKSZ2NUXQJZE2IKNWP5PAL3TN2NNHD3ATZUQHGKKYMSSM3LQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.066874873 (524h17m24.528967356s) [noise=true , address=145.79.227.7:4294 , piece_id=H4ZNVIXORNO4MTXOVYDQZI4IJE4VXHZWC6JI35YKOR3UESSXJNTQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.067301676 (524h17m24.529394159s) [noise=true , address=45.91.116.9:4262 , piece_id=X53JZGRLZLX53UZLLCXXUQCFSQD2FO76X74OM45G26HNDU55UXXQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.067705735 (524h17m24.529798218s) [noise=true , address=145.79.227.4:4187 , piece_id=TQVPL6ZA6LHKAJVL6XWSYASZD3MCKKV3553CRFD72XWTVWY62STQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.069183369 (524h17m24.531275862s) [noise=true , address=145.79.227.5:4222 , piece_id=C2JSPN2QVSGSXG6ZW3ZMOFX5YFNN65XY7E62N5OEHOOMK3NSO4KQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.070534782 (524h17m24.532627265s) [noise=true , address=77.90.31.11:4555 , piece_id=U5RYZSD643KSUI4XLL26EAN6FQRPMG62GQMMCQRPEEG2XKSNBIZQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.072356982 (524h17m24.534449466s) [noise=true , address=145.79.227.11:4730 , piece_id=VO2DSM5LT43QAFPWOY3W3PVEJ44GZJWTCWG5QRQQFB363DFVFZDA, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.073961167 (524h17m24.53605364s) [noise=true , address=45.91.116.4:4940 , piece_id=3MCSUDYN73NIVJTGKPOVY6N3MALB3GBXI7I5NAX4HXPDHVMQ52BA, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.075716960 (524h17m24.537809433s) [noise=true , address=145.79.227.16:4910 , piece_id=BIHEJDH7RPI4UCPUNV453QHSEVCLA54PSNNQB2II37XZDRXFZNWQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.079266419 (524h17m24.541358902s) [noise=true , address=145.79.227.14:4838 , piece_id=6K6A4CNEHGB3X6DEBLF5KHLUT4GE6BQBW6IE3RHBML2NWPN255AA, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.080943032 (524h17m24.543035505s) [noise=true , address=45.91.116.12:4226 , piece_id=DRXRGO7OXEKVS5ACL6FC7RKLGBBD6KNI7X6E5DM5D5VV5PEKN7FQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.081340479 (524h17m24.543432962s) [noise=true , address=45.91.116.8:4045 , piece_id=EDXBNLKKRZMT6MKK2KOIZM7DDKA3IHB6UMCXX4YKZF7JY2T3S5CQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.083616223 (524h17m24.545708706s) [noise=true , address=145.79.227.14:4840 , piece_id=B4ZVPRMVH7Y3TCWCLPO7DWRVMJL7VSI73CIT4PEHCMTIEURQKB2Q, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.083633946 (524h17m24.545726429s) [noise=true , address=45.91.116.14:4189 , piece_id=UMQRQYPZ7SWUCO2N7JDCEVI5U5P36FDL3SUM6HIYJVWVOZTWNXQQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.085493247 (524h17m24.54758573s) [noise=true , address=145.79.227.17:4969 , piece_id=R5HIP33QW5HLICARFWDUOW7I75YTDHZTJCIML5X52VKHLPL3J4YQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:40.727263388 (524h17m26.189355871s) [upload=461218]: Next returned nil segment Oct 15 09:48:40.727310076 (524h17m26.189402549s) [upload=461218]: Waiting for error group managing segments... Oct 15 09:48:41.004514306 (524h17m26.466606789s) [noise=true , address=145.79.227.7:4294 , piece_id=H4ZNVIXORNO4MTXOVYDQZI4IJE4VXHZWC6JI35YKOR3UESSXJNTQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.006116557 (524h17m26.46820904s) [noise=true , address=145.79.227.5:4222 , piece_id=C2JSPN2QVSGSXG6ZW3ZMOFX5YFNN65XY7E62N5OEHOOMK3NSO4KQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.012963184 (524h17m26.475055667s) [noise=true , address=145.79.227.16:4910 , piece_id=BIHEJDH7RPI4UCPUNV453QHSEVCLA54PSNNQB2II37XZDRXFZNWQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.012994684 (524h17m26.475087167s) [noise=true , address=145.79.227.14:4840 , piece_id=B4ZVPRMVH7Y3TCWCLPO7DWRVMJL7VSI73CIT4PEHCMTIEURQKB2Q, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.016649956 (524h17m26.478742439s) [noise=true , address=145.79.227.11:4730 , piece_id=VO2DSM5LT43QAFPWOY3W3PVEJ44GZJWTCWG5QRQQFB363DFVFZDA, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.019248864 (524h17m26.481341347s) [noise=true , address=145.79.227.17:4969 , piece_id=R5HIP33QW5HLICARFWDUOW7I75YTDHZTJCIML5X52VKHLPL3J4YQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.075991585 (524h17m26.538084068s) [noise=true , address=145.79.227.4:4187 , piece_id=TQVPL6ZA6LHKAJVL6XWSYASZD3MCKKV3553CRFD72XWTVWY62STQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.118655146 (524h17m26.580747619s) [noise=true , address=145.79.227.14:4838 , piece_id=6K6A4CNEHGB3X6DEBLF5KHLUT4GE6BQBW6IE3RHBML2NWPN255AA, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.285934795 (524h17m26.748027278s) [noise=true , address=77.90.31.11:4555 , piece_id=U5RYZSD643KSUI4XLL26EAN6FQRPMG62GQMMCQRPEEG2XKSNBIZQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.744525465 (524h17m27.206617948s) [noise=true , address=45.91.116.9:4262 , piece_id=X53JZGRLZLX53UZLLCXXUQCFSQD2FO76X74OM45G26HNDU55UXXQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:42.218961989 (524h17m27.681054462s) [noise=true , address=45.91.116.11:4117 , piece_id=VH7ZJKSZ2NUXQJZE2IKNWP5PAL3TN2NNHD3ATZUQHGKKYMSSM3LQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:42.313638675 (524h17m27.775731158s) [noise=true , address=45.91.116.14:4189 , piece_id=UMQRQYPZ7SWUCO2N7JDCEVI5U5P36FDL3SUM6HIYJVWVOZTWNXQQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:42.434283559 (524h17m27.896376053s) [noise=true , address=45.91.116.8:4045 , piece_id=EDXBNLKKRZMT6MKK2KOIZM7DDKA3IHB6UMCXX4YKZF7JY2T3S5CQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:42.434321832 (524h17m27.896414305s) [seg_pos={0 0}, upload=461218]: Segment reached optimal threshold of 13 pieces. Oct 15 09:48:42.434605041 (524h17m27.896697534s) [noise=true , address=45.91.116.5:4299 , piece_id=4ARMXXE5LBRE2OZQEIITELKPTW4TISZWTD6XZGI4UJV7UCLUXAEQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX): protocol: expected piece hash; context canceled; EOF Oct 15 09:48:42.434666138 (524h17m27.896758621s) [noise=true , address=45.91.116.5:4299 , piece_id=4ARMXXE5LBRE2OZQEIITELKPTW4TISZWTD6XZGI4UJV7UCLUXAEQ, seg_pos={0 0}, upload=461218]: Long tail cancellation due to optimal threshold Oct 15 09:48:42.434692849 (524h17m27.896785432s) [noise=true , address=45.91.116.12:4226 , piece_id=DRXRGO7OXEKVS5ACL6FC7RKLGBBD6KNI7X6E5DM5D5VV5PEKN7FQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1): protocol: expected piece hash; context canceled; EOF Oct 15 09:48:42.434706184 (524h17m27.896798657s) [noise=true , address=45.91.116.12:4226 , piece_id=DRXRGO7OXEKVS5ACL6FC7RKLGBBD6KNI7X6E5DM5D5VV5PEKN7FQ, seg_pos={0 0}, upload=461218]: Long tail cancellation due to optimal threshold Oct 15 09:48:42.434718318 (524h17m27.896810911s) [noise=true , address=45.91.116.4:4940 , piece_id=3MCSUDYN73NIVJTGKPOVY6N3MALB3GBXI7I5NAX4HXPDHVMQ52BA, seg_pos={0 0}, upload=461218]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe): protocol: expected piece hash; context canceled; EOF Oct 15 09:48:42.434728447 (524h17m27.89682093s) [noise=true , address=45.91.116.4:4940 , piece_id=3MCSUDYN73NIVJTGKPOVY6N3MALB3GBXI7I5NAX4HXPDHVMQ52BA, seg_pos={0 0}, upload=461218]: Long tail cancellation due to optimal threshold Oct 15 09:48:42.434760458 (524h17m27.896852931s) [seg_pos={0 0}, upload=461218]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 15 09:48:42.434866900 (524h17m27.896959384s) [upload=461218]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 15 09:48:42.434873884 (524h17m27.896966367s) [upload=461218]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 15 09:48:42.760633614 (524h17m28.222726087s) [upload=461218]: Done uploading segments... Oct 15 09:48:42.760655435 (524h17m28.222747918s) [upload=461218]: Upload finished. err: <nil> Duration: 9.437322715s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.......X...X................................................] duration: 6.173974218s FAIL (2025-10-15 10:25:14 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, Piece ID: VH7ZJKSZ2NUXQJZE2IKNWP5PAL3TN2NNHD3ATZUQHGKKYMSSM3LQ) context deadline exceeded; piecestore: (Node ID: 1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N, Piece ID: UMQRQYPZ7SWUCO2N7JDCEVI5U5P36FDL3SUM6HIYJVWVOZTWNXQQ) context deadline exceeded ======LOGS====== Oct 15 09:48:38.801058027 (524h17m24.26315051s) [upload=461218]: Starting upload Oct 15 09:48:38.801138119 (524h17m24.263230592s) [upload=461218]: Done starting upload Oct 15 09:48:38.801288545 (524h17m24.263381028s) [upload=461218]: Uploading segments... Oct 15 09:48:38.801442509 (524h17m24.263535002s) [upload=461218]: Got next segment. Inline: false Oct 15 09:48:38.801486663 (524h17m24.263579136s) [upload=461218]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 15 09:48:38.801493245 (524h17m24.263585728s) [upload=461218]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 15 09:48:39.066131918 (524h17m24.528224401s) [seg_pos={0 0}, upload=461218]: Begin upload segment... Oct 15 09:48:39.066205549 (524h17m24.528298022s) [seg_pos={0 0}, upload=461218]: Done begin upload segment. Oct 15 09:48:39.066243090 (524h17m24.528335573s) [noise=true , address=45.91.116.5:4299 , piece_id=4ARMXXE5LBRE2OZQEIITELKPTW4TISZWTD6XZGI4UJV7UCLUXAEQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.066254431 (524h17m24.528346915s) [noise=true , address=45.91.116.11:4117 , piece_id=VH7ZJKSZ2NUXQJZE2IKNWP5PAL3TN2NNHD3ATZUQHGKKYMSSM3LQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.066874873 (524h17m24.528967356s) [noise=true , address=145.79.227.7:4294 , piece_id=H4ZNVIXORNO4MTXOVYDQZI4IJE4VXHZWC6JI35YKOR3UESSXJNTQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.067301676 (524h17m24.529394159s) [noise=true , address=45.91.116.9:4262 , piece_id=X53JZGRLZLX53UZLLCXXUQCFSQD2FO76X74OM45G26HNDU55UXXQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.067705735 (524h17m24.529798218s) [noise=true , address=145.79.227.4:4187 , piece_id=TQVPL6ZA6LHKAJVL6XWSYASZD3MCKKV3553CRFD72XWTVWY62STQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.069183369 (524h17m24.531275862s) [noise=true , address=145.79.227.5:4222 , piece_id=C2JSPN2QVSGSXG6ZW3ZMOFX5YFNN65XY7E62N5OEHOOMK3NSO4KQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.070534782 (524h17m24.532627265s) [noise=true , address=77.90.31.11:4555 , piece_id=U5RYZSD643KSUI4XLL26EAN6FQRPMG62GQMMCQRPEEG2XKSNBIZQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.072356982 (524h17m24.534449466s) [noise=true , address=145.79.227.11:4730 , piece_id=VO2DSM5LT43QAFPWOY3W3PVEJ44GZJWTCWG5QRQQFB363DFVFZDA, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.073961167 (524h17m24.53605364s) [noise=true , address=45.91.116.4:4940 , piece_id=3MCSUDYN73NIVJTGKPOVY6N3MALB3GBXI7I5NAX4HXPDHVMQ52BA, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.075716960 (524h17m24.537809433s) [noise=true , address=145.79.227.16:4910 , piece_id=BIHEJDH7RPI4UCPUNV453QHSEVCLA54PSNNQB2II37XZDRXFZNWQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.079266419 (524h17m24.541358902s) [noise=true , address=145.79.227.14:4838 , piece_id=6K6A4CNEHGB3X6DEBLF5KHLUT4GE6BQBW6IE3RHBML2NWPN255AA, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.080943032 (524h17m24.543035505s) [noise=true , address=45.91.116.12:4226 , piece_id=DRXRGO7OXEKVS5ACL6FC7RKLGBBD6KNI7X6E5DM5D5VV5PEKN7FQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.081340479 (524h17m24.543432962s) [noise=true , address=45.91.116.8:4045 , piece_id=EDXBNLKKRZMT6MKK2KOIZM7DDKA3IHB6UMCXX4YKZF7JY2T3S5CQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.083616223 (524h17m24.545708706s) [noise=true , address=145.79.227.14:4840 , piece_id=B4ZVPRMVH7Y3TCWCLPO7DWRVMJL7VSI73CIT4PEHCMTIEURQKB2Q, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.083633946 (524h17m24.545726429s) [noise=true , address=45.91.116.14:4189 , piece_id=UMQRQYPZ7SWUCO2N7JDCEVI5U5P36FDL3SUM6HIYJVWVOZTWNXQQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.085493247 (524h17m24.54758573s) [noise=true , address=145.79.227.17:4969 , piece_id=R5HIP33QW5HLICARFWDUOW7I75YTDHZTJCIML5X52VKHLPL3J4YQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:40.727263388 (524h17m26.189355871s) [upload=461218]: Next returned nil segment Oct 15 09:48:40.727310076 (524h17m26.189402549s) [upload=461218]: Waiting for error group managing segments... Oct 15 09:48:41.004514306 (524h17m26.466606789s) [noise=true , address=145.79.227.7:4294 , piece_id=H4ZNVIXORNO4MTXOVYDQZI4IJE4VXHZWC6JI35YKOR3UESSXJNTQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.006116557 (524h17m26.46820904s) [noise=true , address=145.79.227.5:4222 , piece_id=C2JSPN2QVSGSXG6ZW3ZMOFX5YFNN65XY7E62N5OEHOOMK3NSO4KQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.012963184 (524h17m26.475055667s) [noise=true , address=145.79.227.16:4910 , piece_id=BIHEJDH7RPI4UCPUNV453QHSEVCLA54PSNNQB2II37XZDRXFZNWQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.012994684 (524h17m26.475087167s) [noise=true , address=145.79.227.14:4840 , piece_id=B4ZVPRMVH7Y3TCWCLPO7DWRVMJL7VSI73CIT4PEHCMTIEURQKB2Q, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.016649956 (524h17m26.478742439s) [noise=true , address=145.79.227.11:4730 , piece_id=VO2DSM5LT43QAFPWOY3W3PVEJ44GZJWTCWG5QRQQFB363DFVFZDA, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.019248864 (524h17m26.481341347s) [noise=true , address=145.79.227.17:4969 , piece_id=R5HIP33QW5HLICARFWDUOW7I75YTDHZTJCIML5X52VKHLPL3J4YQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.075991585 (524h17m26.538084068s) [noise=true , address=145.79.227.4:4187 , piece_id=TQVPL6ZA6LHKAJVL6XWSYASZD3MCKKV3553CRFD72XWTVWY62STQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.118655146 (524h17m26.580747619s) [noise=true , address=145.79.227.14:4838 , piece_id=6K6A4CNEHGB3X6DEBLF5KHLUT4GE6BQBW6IE3RHBML2NWPN255AA, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.285934795 (524h17m26.748027278s) [noise=true , address=77.90.31.11:4555 , piece_id=U5RYZSD643KSUI4XLL26EAN6FQRPMG62GQMMCQRPEEG2XKSNBIZQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.744525465 (524h17m27.206617948s) [noise=true , address=45.91.116.9:4262 , piece_id=X53JZGRLZLX53UZLLCXXUQCFSQD2FO76X74OM45G26HNDU55UXXQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:42.218961989 (524h17m27.681054462s) [noise=true , address=45.91.116.11:4117 , piece_id=VH7ZJKSZ2NUXQJZE2IKNWP5PAL3TN2NNHD3ATZUQHGKKYMSSM3LQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:42.313638675 (524h17m27.775731158s) [noise=true , address=45.91.116.14:4189 , piece_id=UMQRQYPZ7SWUCO2N7JDCEVI5U5P36FDL3SUM6HIYJVWVOZTWNXQQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:42.434283559 (524h17m27.896376053s) [noise=true , address=45.91.116.8:4045 , piece_id=EDXBNLKKRZMT6MKK2KOIZM7DDKA3IHB6UMCXX4YKZF7JY2T3S5CQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:42.434321832 (524h17m27.896414305s) [seg_pos={0 0}, upload=461218]: Segment reached optimal threshold of 13 pieces. Oct 15 09:48:42.434605041 (524h17m27.896697534s) [noise=true , address=45.91.116.5:4299 , piece_id=4ARMXXE5LBRE2OZQEIITELKPTW4TISZWTD6XZGI4UJV7UCLUXAEQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX): protocol: expected piece hash; context canceled; EOF Oct 15 09:48:42.434666138 (524h17m27.896758621s) [noise=true , address=45.91.116.5:4299 , piece_id=4ARMXXE5LBRE2OZQEIITELKPTW4TISZWTD6XZGI4UJV7UCLUXAEQ, seg_pos={0 0}, upload=461218]: Long tail cancellation due to optimal threshold Oct 15 09:48:42.434692849 (524h17m27.896785432s) [noise=true , address=45.91.116.12:4226 , piece_id=DRXRGO7OXEKVS5ACL6FC7RKLGBBD6KNI7X6E5DM5D5VV5PEKN7FQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1): protocol: expected piece hash; context canceled; EOF Oct 15 09:48:42.434706184 (524h17m27.896798657s) [noise=true , address=45.91.116.12:4226 , piece_id=DRXRGO7OXEKVS5ACL6FC7RKLGBBD6KNI7X6E5DM5D5VV5PEKN7FQ, seg_pos={0 0}, upload=461218]: Long tail cancellation due to optimal threshold Oct 15 09:48:42.434718318 (524h17m27.896810911s) [noise=true , address=45.91.116.4:4940 , piece_id=3MCSUDYN73NIVJTGKPOVY6N3MALB3GBXI7I5NAX4HXPDHVMQ52BA, seg_pos={0 0}, upload=461218]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe): protocol: expected piece hash; context canceled; EOF Oct 15 09:48:42.434728447 (524h17m27.89682093s) [noise=true , address=45.91.116.4:4940 , piece_id=3MCSUDYN73NIVJTGKPOVY6N3MALB3GBXI7I5NAX4HXPDHVMQ52BA, seg_pos={0 0}, upload=461218]: Long tail cancellation due to optimal threshold Oct 15 09:48:42.434760458 (524h17m27.896852931s) [seg_pos={0 0}, upload=461218]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 15 09:48:42.434866900 (524h17m27.896959384s) [upload=461218]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 15 09:48:42.434873884 (524h17m27.896966367s) [upload=461218]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 15 09:48:42.760633614 (524h17m28.222726087s) [upload=461218]: Done uploading segments... Oct 15 09:48:42.760655435 (524h17m28.222747918s) [upload=461218]: Upload finished. err: <nil> Duration: 6.174008222s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [........X...X...............................................] duration: 5.967393101s FAIL (2025-10-15 10:24:31 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, Piece ID: VH7ZJKSZ2NUXQJZE2IKNWP5PAL3TN2NNHD3ATZUQHGKKYMSSM3LQ) context deadline exceeded; piecestore: (Node ID: 1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N, Piece ID: UMQRQYPZ7SWUCO2N7JDCEVI5U5P36FDL3SUM6HIYJVWVOZTWNXQQ) context deadline exceeded ======LOGS====== Oct 15 09:48:38.801058027 (524h17m24.26315051s) [upload=461218]: Starting upload Oct 15 09:48:38.801138119 (524h17m24.263230592s) [upload=461218]: Done starting upload Oct 15 09:48:38.801288545 (524h17m24.263381028s) [upload=461218]: Uploading segments... Oct 15 09:48:38.801442509 (524h17m24.263535002s) [upload=461218]: Got next segment. Inline: false Oct 15 09:48:38.801486663 (524h17m24.263579136s) [upload=461218]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 15 09:48:38.801493245 (524h17m24.263585728s) [upload=461218]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 15 09:48:39.066131918 (524h17m24.528224401s) [seg_pos={0 0}, upload=461218]: Begin upload segment... Oct 15 09:48:39.066205549 (524h17m24.528298022s) [seg_pos={0 0}, upload=461218]: Done begin upload segment. Oct 15 09:48:39.066243090 (524h17m24.528335573s) [noise=true , address=45.91.116.5:4299 , piece_id=4ARMXXE5LBRE2OZQEIITELKPTW4TISZWTD6XZGI4UJV7UCLUXAEQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.066254431 (524h17m24.528346915s) [noise=true , address=45.91.116.11:4117 , piece_id=VH7ZJKSZ2NUXQJZE2IKNWP5PAL3TN2NNHD3ATZUQHGKKYMSSM3LQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.066874873 (524h17m24.528967356s) [noise=true , address=145.79.227.7:4294 , piece_id=H4ZNVIXORNO4MTXOVYDQZI4IJE4VXHZWC6JI35YKOR3UESSXJNTQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.067301676 (524h17m24.529394159s) [noise=true , address=45.91.116.9:4262 , piece_id=X53JZGRLZLX53UZLLCXXUQCFSQD2FO76X74OM45G26HNDU55UXXQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.067705735 (524h17m24.529798218s) [noise=true , address=145.79.227.4:4187 , piece_id=TQVPL6ZA6LHKAJVL6XWSYASZD3MCKKV3553CRFD72XWTVWY62STQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.069183369 (524h17m24.531275862s) [noise=true , address=145.79.227.5:4222 , piece_id=C2JSPN2QVSGSXG6ZW3ZMOFX5YFNN65XY7E62N5OEHOOMK3NSO4KQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.070534782 (524h17m24.532627265s) [noise=true , address=77.90.31.11:4555 , piece_id=U5RYZSD643KSUI4XLL26EAN6FQRPMG62GQMMCQRPEEG2XKSNBIZQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.072356982 (524h17m24.534449466s) [noise=true , address=145.79.227.11:4730 , piece_id=VO2DSM5LT43QAFPWOY3W3PVEJ44GZJWTCWG5QRQQFB363DFVFZDA, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.073961167 (524h17m24.53605364s) [noise=true , address=45.91.116.4:4940 , piece_id=3MCSUDYN73NIVJTGKPOVY6N3MALB3GBXI7I5NAX4HXPDHVMQ52BA, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.075716960 (524h17m24.537809433s) [noise=true , address=145.79.227.16:4910 , piece_id=BIHEJDH7RPI4UCPUNV453QHSEVCLA54PSNNQB2II37XZDRXFZNWQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.079266419 (524h17m24.541358902s) [noise=true , address=145.79.227.14:4838 , piece_id=6K6A4CNEHGB3X6DEBLF5KHLUT4GE6BQBW6IE3RHBML2NWPN255AA, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.080943032 (524h17m24.543035505s) [noise=true , address=45.91.116.12:4226 , piece_id=DRXRGO7OXEKVS5ACL6FC7RKLGBBD6KNI7X6E5DM5D5VV5PEKN7FQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.081340479 (524h17m24.543432962s) [noise=true , address=45.91.116.8:4045 , piece_id=EDXBNLKKRZMT6MKK2KOIZM7DDKA3IHB6UMCXX4YKZF7JY2T3S5CQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.083616223 (524h17m24.545708706s) [noise=true , address=145.79.227.14:4840 , piece_id=B4ZVPRMVH7Y3TCWCLPO7DWRVMJL7VSI73CIT4PEHCMTIEURQKB2Q, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.083633946 (524h17m24.545726429s) [noise=true , address=45.91.116.14:4189 , piece_id=UMQRQYPZ7SWUCO2N7JDCEVI5U5P36FDL3SUM6HIYJVWVOZTWNXQQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.085493247 (524h17m24.54758573s) [noise=true , address=145.79.227.17:4969 , piece_id=R5HIP33QW5HLICARFWDUOW7I75YTDHZTJCIML5X52VKHLPL3J4YQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:40.727263388 (524h17m26.189355871s) [upload=461218]: Next returned nil segment Oct 15 09:48:40.727310076 (524h17m26.189402549s) [upload=461218]: Waiting for error group managing segments... Oct 15 09:48:41.004514306 (524h17m26.466606789s) [noise=true , address=145.79.227.7:4294 , piece_id=H4ZNVIXORNO4MTXOVYDQZI4IJE4VXHZWC6JI35YKOR3UESSXJNTQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.006116557 (524h17m26.46820904s) [noise=true , address=145.79.227.5:4222 , piece_id=C2JSPN2QVSGSXG6ZW3ZMOFX5YFNN65XY7E62N5OEHOOMK3NSO4KQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.012963184 (524h17m26.475055667s) [noise=true , address=145.79.227.16:4910 , piece_id=BIHEJDH7RPI4UCPUNV453QHSEVCLA54PSNNQB2II37XZDRXFZNWQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.012994684 (524h17m26.475087167s) [noise=true , address=145.79.227.14:4840 , piece_id=B4ZVPRMVH7Y3TCWCLPO7DWRVMJL7VSI73CIT4PEHCMTIEURQKB2Q, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.016649956 (524h17m26.478742439s) [noise=true , address=145.79.227.11:4730 , piece_id=VO2DSM5LT43QAFPWOY3W3PVEJ44GZJWTCWG5QRQQFB363DFVFZDA, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.019248864 (524h17m26.481341347s) [noise=true , address=145.79.227.17:4969 , piece_id=R5HIP33QW5HLICARFWDUOW7I75YTDHZTJCIML5X52VKHLPL3J4YQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.075991585 (524h17m26.538084068s) [noise=true , address=145.79.227.4:4187 , piece_id=TQVPL6ZA6LHKAJVL6XWSYASZD3MCKKV3553CRFD72XWTVWY62STQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.118655146 (524h17m26.580747619s) [noise=true , address=145.79.227.14:4838 , piece_id=6K6A4CNEHGB3X6DEBLF5KHLUT4GE6BQBW6IE3RHBML2NWPN255AA, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.285934795 (524h17m26.748027278s) [noise=true , address=77.90.31.11:4555 , piece_id=U5RYZSD643KSUI4XLL26EAN6FQRPMG62GQMMCQRPEEG2XKSNBIZQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.744525465 (524h17m27.206617948s) [noise=true , address=45.91.116.9:4262 , piece_id=X53JZGRLZLX53UZLLCXXUQCFSQD2FO76X74OM45G26HNDU55UXXQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:42.218961989 (524h17m27.681054462s) [noise=true , address=45.91.116.11:4117 , piece_id=VH7ZJKSZ2NUXQJZE2IKNWP5PAL3TN2NNHD3ATZUQHGKKYMSSM3LQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:42.313638675 (524h17m27.775731158s) [noise=true , address=45.91.116.14:4189 , piece_id=UMQRQYPZ7SWUCO2N7JDCEVI5U5P36FDL3SUM6HIYJVWVOZTWNXQQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:42.434283559 (524h17m27.896376053s) [noise=true , address=45.91.116.8:4045 , piece_id=EDXBNLKKRZMT6MKK2KOIZM7DDKA3IHB6UMCXX4YKZF7JY2T3S5CQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:42.434321832 (524h17m27.896414305s) [seg_pos={0 0}, upload=461218]: Segment reached optimal threshold of 13 pieces. Oct 15 09:48:42.434605041 (524h17m27.896697534s) [noise=true , address=45.91.116.5:4299 , piece_id=4ARMXXE5LBRE2OZQEIITELKPTW4TISZWTD6XZGI4UJV7UCLUXAEQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX): protocol: expected piece hash; context canceled; EOF Oct 15 09:48:42.434666138 (524h17m27.896758621s) [noise=true , address=45.91.116.5:4299 , piece_id=4ARMXXE5LBRE2OZQEIITELKPTW4TISZWTD6XZGI4UJV7UCLUXAEQ, seg_pos={0 0}, upload=461218]: Long tail cancellation due to optimal threshold Oct 15 09:48:42.434692849 (524h17m27.896785432s) [noise=true , address=45.91.116.12:4226 , piece_id=DRXRGO7OXEKVS5ACL6FC7RKLGBBD6KNI7X6E5DM5D5VV5PEKN7FQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1): protocol: expected piece hash; context canceled; EOF Oct 15 09:48:42.434706184 (524h17m27.896798657s) [noise=true , address=45.91.116.12:4226 , piece_id=DRXRGO7OXEKVS5ACL6FC7RKLGBBD6KNI7X6E5DM5D5VV5PEKN7FQ, seg_pos={0 0}, upload=461218]: Long tail cancellation due to optimal threshold Oct 15 09:48:42.434718318 (524h17m27.896810911s) [noise=true , address=45.91.116.4:4940 , piece_id=3MCSUDYN73NIVJTGKPOVY6N3MALB3GBXI7I5NAX4HXPDHVMQ52BA, seg_pos={0 0}, upload=461218]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe): protocol: expected piece hash; context canceled; EOF Oct 15 09:48:42.434728447 (524h17m27.89682093s) [noise=true , address=45.91.116.4:4940 , piece_id=3MCSUDYN73NIVJTGKPOVY6N3MALB3GBXI7I5NAX4HXPDHVMQ52BA, seg_pos={0 0}, upload=461218]: Long tail cancellation due to optimal threshold Oct 15 09:48:42.434760458 (524h17m27.896852931s) [seg_pos={0 0}, upload=461218]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 15 09:48:42.434866900 (524h17m27.896959384s) [upload=461218]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 15 09:48:42.434873884 (524h17m27.896966367s) [upload=461218]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 15 09:48:42.760633614 (524h17m28.222726087s) [upload=461218]: Done uploading segments... Oct 15 09:48:42.760655435 (524h17m28.222747918s) [upload=461218]: Upload finished. err: <nil> Duration: 5.967426433s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.........X...X..............................................] duration: 6.042883456s FAIL (2025-10-15 10:23:46 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, Piece ID: VH7ZJKSZ2NUXQJZE2IKNWP5PAL3TN2NNHD3ATZUQHGKKYMSSM3LQ) context deadline exceeded; piecestore: (Node ID: 1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N, Piece ID: UMQRQYPZ7SWUCO2N7JDCEVI5U5P36FDL3SUM6HIYJVWVOZTWNXQQ) context deadline exceeded ======LOGS====== Oct 15 09:48:38.801058027 (524h17m24.26315051s) [upload=461218]: Starting upload Oct 15 09:48:38.801138119 (524h17m24.263230592s) [upload=461218]: Done starting upload Oct 15 09:48:38.801288545 (524h17m24.263381028s) [upload=461218]: Uploading segments... Oct 15 09:48:38.801442509 (524h17m24.263535002s) [upload=461218]: Got next segment. Inline: false Oct 15 09:48:38.801486663 (524h17m24.263579136s) [upload=461218]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 15 09:48:38.801493245 (524h17m24.263585728s) [upload=461218]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 15 09:48:39.066131918 (524h17m24.528224401s) [seg_pos={0 0}, upload=461218]: Begin upload segment... Oct 15 09:48:39.066205549 (524h17m24.528298022s) [seg_pos={0 0}, upload=461218]: Done begin upload segment. Oct 15 09:48:39.066243090 (524h17m24.528335573s) [noise=true , address=45.91.116.5:4299 , piece_id=4ARMXXE5LBRE2OZQEIITELKPTW4TISZWTD6XZGI4UJV7UCLUXAEQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.066254431 (524h17m24.528346915s) [noise=true , address=45.91.116.11:4117 , piece_id=VH7ZJKSZ2NUXQJZE2IKNWP5PAL3TN2NNHD3ATZUQHGKKYMSSM3LQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.066874873 (524h17m24.528967356s) [noise=true , address=145.79.227.7:4294 , piece_id=H4ZNVIXORNO4MTXOVYDQZI4IJE4VXHZWC6JI35YKOR3UESSXJNTQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.067301676 (524h17m24.529394159s) [noise=true , address=45.91.116.9:4262 , piece_id=X53JZGRLZLX53UZLLCXXUQCFSQD2FO76X74OM45G26HNDU55UXXQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.067705735 (524h17m24.529798218s) [noise=true , address=145.79.227.4:4187 , piece_id=TQVPL6ZA6LHKAJVL6XWSYASZD3MCKKV3553CRFD72XWTVWY62STQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.069183369 (524h17m24.531275862s) [noise=true , address=145.79.227.5:4222 , piece_id=C2JSPN2QVSGSXG6ZW3ZMOFX5YFNN65XY7E62N5OEHOOMK3NSO4KQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.070534782 (524h17m24.532627265s) [noise=true , address=77.90.31.11:4555 , piece_id=U5RYZSD643KSUI4XLL26EAN6FQRPMG62GQMMCQRPEEG2XKSNBIZQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.072356982 (524h17m24.534449466s) [noise=true , address=145.79.227.11:4730 , piece_id=VO2DSM5LT43QAFPWOY3W3PVEJ44GZJWTCWG5QRQQFB363DFVFZDA, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.073961167 (524h17m24.53605364s) [noise=true , address=45.91.116.4:4940 , piece_id=3MCSUDYN73NIVJTGKPOVY6N3MALB3GBXI7I5NAX4HXPDHVMQ52BA, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.075716960 (524h17m24.537809433s) [noise=true , address=145.79.227.16:4910 , piece_id=BIHEJDH7RPI4UCPUNV453QHSEVCLA54PSNNQB2II37XZDRXFZNWQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.079266419 (524h17m24.541358902s) [noise=true , address=145.79.227.14:4838 , piece_id=6K6A4CNEHGB3X6DEBLF5KHLUT4GE6BQBW6IE3RHBML2NWPN255AA, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.080943032 (524h17m24.543035505s) [noise=true , address=45.91.116.12:4226 , piece_id=DRXRGO7OXEKVS5ACL6FC7RKLGBBD6KNI7X6E5DM5D5VV5PEKN7FQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.081340479 (524h17m24.543432962s) [noise=true , address=45.91.116.8:4045 , piece_id=EDXBNLKKRZMT6MKK2KOIZM7DDKA3IHB6UMCXX4YKZF7JY2T3S5CQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.083616223 (524h17m24.545708706s) [noise=true , address=145.79.227.14:4840 , piece_id=B4ZVPRMVH7Y3TCWCLPO7DWRVMJL7VSI73CIT4PEHCMTIEURQKB2Q, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.083633946 (524h17m24.545726429s) [noise=true , address=45.91.116.14:4189 , piece_id=UMQRQYPZ7SWUCO2N7JDCEVI5U5P36FDL3SUM6HIYJVWVOZTWNXQQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.085493247 (524h17m24.54758573s) [noise=true , address=145.79.227.17:4969 , piece_id=R5HIP33QW5HLICARFWDUOW7I75YTDHZTJCIML5X52VKHLPL3J4YQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:40.727263388 (524h17m26.189355871s) [upload=461218]: Next returned nil segment Oct 15 09:48:40.727310076 (524h17m26.189402549s) [upload=461218]: Waiting for error group managing segments... Oct 15 09:48:41.004514306 (524h17m26.466606789s) [noise=true , address=145.79.227.7:4294 , piece_id=H4ZNVIXORNO4MTXOVYDQZI4IJE4VXHZWC6JI35YKOR3UESSXJNTQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.006116557 (524h17m26.46820904s) [noise=true , address=145.79.227.5:4222 , piece_id=C2JSPN2QVSGSXG6ZW3ZMOFX5YFNN65XY7E62N5OEHOOMK3NSO4KQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.012963184 (524h17m26.475055667s) [noise=true , address=145.79.227.16:4910 , piece_id=BIHEJDH7RPI4UCPUNV453QHSEVCLA54PSNNQB2II37XZDRXFZNWQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.012994684 (524h17m26.475087167s) [noise=true , address=145.79.227.14:4840 , piece_id=B4ZVPRMVH7Y3TCWCLPO7DWRVMJL7VSI73CIT4PEHCMTIEURQKB2Q, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.016649956 (524h17m26.478742439s) [noise=true , address=145.79.227.11:4730 , piece_id=VO2DSM5LT43QAFPWOY3W3PVEJ44GZJWTCWG5QRQQFB363DFVFZDA, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.019248864 (524h17m26.481341347s) [noise=true , address=145.79.227.17:4969 , piece_id=R5HIP33QW5HLICARFWDUOW7I75YTDHZTJCIML5X52VKHLPL3J4YQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.075991585 (524h17m26.538084068s) [noise=true , address=145.79.227.4:4187 , piece_id=TQVPL6ZA6LHKAJVL6XWSYASZD3MCKKV3553CRFD72XWTVWY62STQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.118655146 (524h17m26.580747619s) [noise=true , address=145.79.227.14:4838 , piece_id=6K6A4CNEHGB3X6DEBLF5KHLUT4GE6BQBW6IE3RHBML2NWPN255AA, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.285934795 (524h17m26.748027278s) [noise=true , address=77.90.31.11:4555 , piece_id=U5RYZSD643KSUI4XLL26EAN6FQRPMG62GQMMCQRPEEG2XKSNBIZQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.744525465 (524h17m27.206617948s) [noise=true , address=45.91.116.9:4262 , piece_id=X53JZGRLZLX53UZLLCXXUQCFSQD2FO76X74OM45G26HNDU55UXXQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:42.218961989 (524h17m27.681054462s) [noise=true , address=45.91.116.11:4117 , piece_id=VH7ZJKSZ2NUXQJZE2IKNWP5PAL3TN2NNHD3ATZUQHGKKYMSSM3LQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:42.313638675 (524h17m27.775731158s) [noise=true , address=45.91.116.14:4189 , piece_id=UMQRQYPZ7SWUCO2N7JDCEVI5U5P36FDL3SUM6HIYJVWVOZTWNXQQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:42.434283559 (524h17m27.896376053s) [noise=true , address=45.91.116.8:4045 , piece_id=EDXBNLKKRZMT6MKK2KOIZM7DDKA3IHB6UMCXX4YKZF7JY2T3S5CQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:42.434321832 (524h17m27.896414305s) [seg_pos={0 0}, upload=461218]: Segment reached optimal threshold of 13 pieces. Oct 15 09:48:42.434605041 (524h17m27.896697534s) [noise=true , address=45.91.116.5:4299 , piece_id=4ARMXXE5LBRE2OZQEIITELKPTW4TISZWTD6XZGI4UJV7UCLUXAEQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX): protocol: expected piece hash; context canceled; EOF Oct 15 09:48:42.434666138 (524h17m27.896758621s) [noise=true , address=45.91.116.5:4299 , piece_id=4ARMXXE5LBRE2OZQEIITELKPTW4TISZWTD6XZGI4UJV7UCLUXAEQ, seg_pos={0 0}, upload=461218]: Long tail cancellation due to optimal threshold Oct 15 09:48:42.434692849 (524h17m27.896785432s) [noise=true , address=45.91.116.12:4226 , piece_id=DRXRGO7OXEKVS5ACL6FC7RKLGBBD6KNI7X6E5DM5D5VV5PEKN7FQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1): protocol: expected piece hash; context canceled; EOF Oct 15 09:48:42.434706184 (524h17m27.896798657s) [noise=true , address=45.91.116.12:4226 , piece_id=DRXRGO7OXEKVS5ACL6FC7RKLGBBD6KNI7X6E5DM5D5VV5PEKN7FQ, seg_pos={0 0}, upload=461218]: Long tail cancellation due to optimal threshold Oct 15 09:48:42.434718318 (524h17m27.896810911s) [noise=true , address=45.91.116.4:4940 , piece_id=3MCSUDYN73NIVJTGKPOVY6N3MALB3GBXI7I5NAX4HXPDHVMQ52BA, seg_pos={0 0}, upload=461218]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe): protocol: expected piece hash; context canceled; EOF Oct 15 09:48:42.434728447 (524h17m27.89682093s) [noise=true , address=45.91.116.4:4940 , piece_id=3MCSUDYN73NIVJTGKPOVY6N3MALB3GBXI7I5NAX4HXPDHVMQ52BA, seg_pos={0 0}, upload=461218]: Long tail cancellation due to optimal threshold Oct 15 09:48:42.434760458 (524h17m27.896852931s) [seg_pos={0 0}, upload=461218]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 15 09:48:42.434866900 (524h17m27.896959384s) [upload=461218]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 15 09:48:42.434873884 (524h17m27.896966367s) [upload=461218]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 15 09:48:42.760633614 (524h17m28.222726087s) [upload=461218]: Done uploading segments... Oct 15 09:48:42.760655435 (524h17m28.222747918s) [upload=461218]: Upload finished. err: <nil> Duration: 6.043003715s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..........X...X.............................................] duration: 6.108811623s FAIL (2025-10-15 10:22:45 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, Piece ID: VH7ZJKSZ2NUXQJZE2IKNWP5PAL3TN2NNHD3ATZUQHGKKYMSSM3LQ) context deadline exceeded; piecestore: (Node ID: 1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N, Piece ID: UMQRQYPZ7SWUCO2N7JDCEVI5U5P36FDL3SUM6HIYJVWVOZTWNXQQ) context deadline exceeded ======LOGS====== Oct 15 09:48:38.801058027 (524h17m24.26315051s) [upload=461218]: Starting upload Oct 15 09:48:38.801138119 (524h17m24.263230592s) [upload=461218]: Done starting upload Oct 15 09:48:38.801288545 (524h17m24.263381028s) [upload=461218]: Uploading segments... Oct 15 09:48:38.801442509 (524h17m24.263535002s) [upload=461218]: Got next segment. Inline: false Oct 15 09:48:38.801486663 (524h17m24.263579136s) [upload=461218]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 15 09:48:38.801493245 (524h17m24.263585728s) [upload=461218]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 15 09:48:39.066131918 (524h17m24.528224401s) [seg_pos={0 0}, upload=461218]: Begin upload segment... Oct 15 09:48:39.066205549 (524h17m24.528298022s) [seg_pos={0 0}, upload=461218]: Done begin upload segment. Oct 15 09:48:39.066243090 (524h17m24.528335573s) [noise=true , address=45.91.116.5:4299 , piece_id=4ARMXXE5LBRE2OZQEIITELKPTW4TISZWTD6XZGI4UJV7UCLUXAEQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.066254431 (524h17m24.528346915s) [noise=true , address=45.91.116.11:4117 , piece_id=VH7ZJKSZ2NUXQJZE2IKNWP5PAL3TN2NNHD3ATZUQHGKKYMSSM3LQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.066874873 (524h17m24.528967356s) [noise=true , address=145.79.227.7:4294 , piece_id=H4ZNVIXORNO4MTXOVYDQZI4IJE4VXHZWC6JI35YKOR3UESSXJNTQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.067301676 (524h17m24.529394159s) [noise=true , address=45.91.116.9:4262 , piece_id=X53JZGRLZLX53UZLLCXXUQCFSQD2FO76X74OM45G26HNDU55UXXQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.067705735 (524h17m24.529798218s) [noise=true , address=145.79.227.4:4187 , piece_id=TQVPL6ZA6LHKAJVL6XWSYASZD3MCKKV3553CRFD72XWTVWY62STQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.069183369 (524h17m24.531275862s) [noise=true , address=145.79.227.5:4222 , piece_id=C2JSPN2QVSGSXG6ZW3ZMOFX5YFNN65XY7E62N5OEHOOMK3NSO4KQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.070534782 (524h17m24.532627265s) [noise=true , address=77.90.31.11:4555 , piece_id=U5RYZSD643KSUI4XLL26EAN6FQRPMG62GQMMCQRPEEG2XKSNBIZQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.072356982 (524h17m24.534449466s) [noise=true , address=145.79.227.11:4730 , piece_id=VO2DSM5LT43QAFPWOY3W3PVEJ44GZJWTCWG5QRQQFB363DFVFZDA, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.073961167 (524h17m24.53605364s) [noise=true , address=45.91.116.4:4940 , piece_id=3MCSUDYN73NIVJTGKPOVY6N3MALB3GBXI7I5NAX4HXPDHVMQ52BA, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.075716960 (524h17m24.537809433s) [noise=true , address=145.79.227.16:4910 , piece_id=BIHEJDH7RPI4UCPUNV453QHSEVCLA54PSNNQB2II37XZDRXFZNWQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.079266419 (524h17m24.541358902s) [noise=true , address=145.79.227.14:4838 , piece_id=6K6A4CNEHGB3X6DEBLF5KHLUT4GE6BQBW6IE3RHBML2NWPN255AA, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.080943032 (524h17m24.543035505s) [noise=true , address=45.91.116.12:4226 , piece_id=DRXRGO7OXEKVS5ACL6FC7RKLGBBD6KNI7X6E5DM5D5VV5PEKN7FQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.081340479 (524h17m24.543432962s) [noise=true , address=45.91.116.8:4045 , piece_id=EDXBNLKKRZMT6MKK2KOIZM7DDKA3IHB6UMCXX4YKZF7JY2T3S5CQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.083616223 (524h17m24.545708706s) [noise=true , address=145.79.227.14:4840 , piece_id=B4ZVPRMVH7Y3TCWCLPO7DWRVMJL7VSI73CIT4PEHCMTIEURQKB2Q, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.083633946 (524h17m24.545726429s) [noise=true , address=45.91.116.14:4189 , piece_id=UMQRQYPZ7SWUCO2N7JDCEVI5U5P36FDL3SUM6HIYJVWVOZTWNXQQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.085493247 (524h17m24.54758573s) [noise=true , address=145.79.227.17:4969 , piece_id=R5HIP33QW5HLICARFWDUOW7I75YTDHZTJCIML5X52VKHLPL3J4YQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:40.727263388 (524h17m26.189355871s) [upload=461218]: Next returned nil segment Oct 15 09:48:40.727310076 (524h17m26.189402549s) [upload=461218]: Waiting for error group managing segments... Oct 15 09:48:41.004514306 (524h17m26.466606789s) [noise=true , address=145.79.227.7:4294 , piece_id=H4ZNVIXORNO4MTXOVYDQZI4IJE4VXHZWC6JI35YKOR3UESSXJNTQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.006116557 (524h17m26.46820904s) [noise=true , address=145.79.227.5:4222 , piece_id=C2JSPN2QVSGSXG6ZW3ZMOFX5YFNN65XY7E62N5OEHOOMK3NSO4KQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.012963184 (524h17m26.475055667s) [noise=true , address=145.79.227.16:4910 , piece_id=BIHEJDH7RPI4UCPUNV453QHSEVCLA54PSNNQB2II37XZDRXFZNWQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.012994684 (524h17m26.475087167s) [noise=true , address=145.79.227.14:4840 , piece_id=B4ZVPRMVH7Y3TCWCLPO7DWRVMJL7VSI73CIT4PEHCMTIEURQKB2Q, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.016649956 (524h17m26.478742439s) [noise=true , address=145.79.227.11:4730 , piece_id=VO2DSM5LT43QAFPWOY3W3PVEJ44GZJWTCWG5QRQQFB363DFVFZDA, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.019248864 (524h17m26.481341347s) [noise=true , address=145.79.227.17:4969 , piece_id=R5HIP33QW5HLICARFWDUOW7I75YTDHZTJCIML5X52VKHLPL3J4YQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.075991585 (524h17m26.538084068s) [noise=true , address=145.79.227.4:4187 , piece_id=TQVPL6ZA6LHKAJVL6XWSYASZD3MCKKV3553CRFD72XWTVWY62STQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.118655146 (524h17m26.580747619s) [noise=true , address=145.79.227.14:4838 , piece_id=6K6A4CNEHGB3X6DEBLF5KHLUT4GE6BQBW6IE3RHBML2NWPN255AA, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.285934795 (524h17m26.748027278s) [noise=true , address=77.90.31.11:4555 , piece_id=U5RYZSD643KSUI4XLL26EAN6FQRPMG62GQMMCQRPEEG2XKSNBIZQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.744525465 (524h17m27.206617948s) [noise=true , address=45.91.116.9:4262 , piece_id=X53JZGRLZLX53UZLLCXXUQCFSQD2FO76X74OM45G26HNDU55UXXQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:42.218961989 (524h17m27.681054462s) [noise=true , address=45.91.116.11:4117 , piece_id=VH7ZJKSZ2NUXQJZE2IKNWP5PAL3TN2NNHD3ATZUQHGKKYMSSM3LQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:42.313638675 (524h17m27.775731158s) [noise=true , address=45.91.116.14:4189 , piece_id=UMQRQYPZ7SWUCO2N7JDCEVI5U5P36FDL3SUM6HIYJVWVOZTWNXQQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:42.434283559 (524h17m27.896376053s) [noise=true , address=45.91.116.8:4045 , piece_id=EDXBNLKKRZMT6MKK2KOIZM7DDKA3IHB6UMCXX4YKZF7JY2T3S5CQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:42.434321832 (524h17m27.896414305s) [seg_pos={0 0}, upload=461218]: Segment reached optimal threshold of 13 pieces. Oct 15 09:48:42.434605041 (524h17m27.896697534s) [noise=true , address=45.91.116.5:4299 , piece_id=4ARMXXE5LBRE2OZQEIITELKPTW4TISZWTD6XZGI4UJV7UCLUXAEQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX): protocol: expected piece hash; context canceled; EOF Oct 15 09:48:42.434666138 (524h17m27.896758621s) [noise=true , address=45.91.116.5:4299 , piece_id=4ARMXXE5LBRE2OZQEIITELKPTW4TISZWTD6XZGI4UJV7UCLUXAEQ, seg_pos={0 0}, upload=461218]: Long tail cancellation due to optimal threshold Oct 15 09:48:42.434692849 (524h17m27.896785432s) [noise=true , address=45.91.116.12:4226 , piece_id=DRXRGO7OXEKVS5ACL6FC7RKLGBBD6KNI7X6E5DM5D5VV5PEKN7FQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1): protocol: expected piece hash; context canceled; EOF Oct 15 09:48:42.434706184 (524h17m27.896798657s) [noise=true , address=45.91.116.12:4226 , piece_id=DRXRGO7OXEKVS5ACL6FC7RKLGBBD6KNI7X6E5DM5D5VV5PEKN7FQ, seg_pos={0 0}, upload=461218]: Long tail cancellation due to optimal threshold Oct 15 09:48:42.434718318 (524h17m27.896810911s) [noise=true , address=45.91.116.4:4940 , piece_id=3MCSUDYN73NIVJTGKPOVY6N3MALB3GBXI7I5NAX4HXPDHVMQ52BA, seg_pos={0 0}, upload=461218]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe): protocol: expected piece hash; context canceled; EOF Oct 15 09:48:42.434728447 (524h17m27.89682093s) [noise=true , address=45.91.116.4:4940 , piece_id=3MCSUDYN73NIVJTGKPOVY6N3MALB3GBXI7I5NAX4HXPDHVMQ52BA, seg_pos={0 0}, upload=461218]: Long tail cancellation due to optimal threshold Oct 15 09:48:42.434760458 (524h17m27.896852931s) [seg_pos={0 0}, upload=461218]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 15 09:48:42.434866900 (524h17m27.896959384s) [upload=461218]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 15 09:48:42.434873884 (524h17m27.896966367s) [upload=461218]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 15 09:48:42.760633614 (524h17m28.222726087s) [upload=461218]: Done uploading segments... Oct 15 09:48:42.760655435 (524h17m28.222747918s) [upload=461218]: Upload finished. err: <nil> Duration: 6.108850457s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...........X...X............................................] duration: 5.932058404s FAIL (2025-10-15 10:22:20 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, Piece ID: VH7ZJKSZ2NUXQJZE2IKNWP5PAL3TN2NNHD3ATZUQHGKKYMSSM3LQ) context deadline exceeded; piecestore: (Node ID: 1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N, Piece ID: UMQRQYPZ7SWUCO2N7JDCEVI5U5P36FDL3SUM6HIYJVWVOZTWNXQQ) context deadline exceeded ======LOGS====== Oct 15 09:48:38.801058027 (524h17m24.26315051s) [upload=461218]: Starting upload Oct 15 09:48:38.801138119 (524h17m24.263230592s) [upload=461218]: Done starting upload Oct 15 09:48:38.801288545 (524h17m24.263381028s) [upload=461218]: Uploading segments... Oct 15 09:48:38.801442509 (524h17m24.263535002s) [upload=461218]: Got next segment. Inline: false Oct 15 09:48:38.801486663 (524h17m24.263579136s) [upload=461218]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 15 09:48:38.801493245 (524h17m24.263585728s) [upload=461218]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 15 09:48:39.066131918 (524h17m24.528224401s) [seg_pos={0 0}, upload=461218]: Begin upload segment... Oct 15 09:48:39.066205549 (524h17m24.528298022s) [seg_pos={0 0}, upload=461218]: Done begin upload segment. Oct 15 09:48:39.066243090 (524h17m24.528335573s) [noise=true , address=45.91.116.5:4299 , piece_id=4ARMXXE5LBRE2OZQEIITELKPTW4TISZWTD6XZGI4UJV7UCLUXAEQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.066254431 (524h17m24.528346915s) [noise=true , address=45.91.116.11:4117 , piece_id=VH7ZJKSZ2NUXQJZE2IKNWP5PAL3TN2NNHD3ATZUQHGKKYMSSM3LQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.066874873 (524h17m24.528967356s) [noise=true , address=145.79.227.7:4294 , piece_id=H4ZNVIXORNO4MTXOVYDQZI4IJE4VXHZWC6JI35YKOR3UESSXJNTQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.067301676 (524h17m24.529394159s) [noise=true , address=45.91.116.9:4262 , piece_id=X53JZGRLZLX53UZLLCXXUQCFSQD2FO76X74OM45G26HNDU55UXXQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.067705735 (524h17m24.529798218s) [noise=true , address=145.79.227.4:4187 , piece_id=TQVPL6ZA6LHKAJVL6XWSYASZD3MCKKV3553CRFD72XWTVWY62STQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.069183369 (524h17m24.531275862s) [noise=true , address=145.79.227.5:4222 , piece_id=C2JSPN2QVSGSXG6ZW3ZMOFX5YFNN65XY7E62N5OEHOOMK3NSO4KQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.070534782 (524h17m24.532627265s) [noise=true , address=77.90.31.11:4555 , piece_id=U5RYZSD643KSUI4XLL26EAN6FQRPMG62GQMMCQRPEEG2XKSNBIZQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.072356982 (524h17m24.534449466s) [noise=true , address=145.79.227.11:4730 , piece_id=VO2DSM5LT43QAFPWOY3W3PVEJ44GZJWTCWG5QRQQFB363DFVFZDA, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.073961167 (524h17m24.53605364s) [noise=true , address=45.91.116.4:4940 , piece_id=3MCSUDYN73NIVJTGKPOVY6N3MALB3GBXI7I5NAX4HXPDHVMQ52BA, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.075716960 (524h17m24.537809433s) [noise=true , address=145.79.227.16:4910 , piece_id=BIHEJDH7RPI4UCPUNV453QHSEVCLA54PSNNQB2II37XZDRXFZNWQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.079266419 (524h17m24.541358902s) [noise=true , address=145.79.227.14:4838 , piece_id=6K6A4CNEHGB3X6DEBLF5KHLUT4GE6BQBW6IE3RHBML2NWPN255AA, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.080943032 (524h17m24.543035505s) [noise=true , address=45.91.116.12:4226 , piece_id=DRXRGO7OXEKVS5ACL6FC7RKLGBBD6KNI7X6E5DM5D5VV5PEKN7FQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.081340479 (524h17m24.543432962s) [noise=true , address=45.91.116.8:4045 , piece_id=EDXBNLKKRZMT6MKK2KOIZM7DDKA3IHB6UMCXX4YKZF7JY2T3S5CQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.083616223 (524h17m24.545708706s) [noise=true , address=145.79.227.14:4840 , piece_id=B4ZVPRMVH7Y3TCWCLPO7DWRVMJL7VSI73CIT4PEHCMTIEURQKB2Q, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.083633946 (524h17m24.545726429s) [noise=true , address=45.91.116.14:4189 , piece_id=UMQRQYPZ7SWUCO2N7JDCEVI5U5P36FDL3SUM6HIYJVWVOZTWNXQQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.085493247 (524h17m24.54758573s) [noise=true , address=145.79.227.17:4969 , piece_id=R5HIP33QW5HLICARFWDUOW7I75YTDHZTJCIML5X52VKHLPL3J4YQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:40.727263388 (524h17m26.189355871s) [upload=461218]: Next returned nil segment Oct 15 09:48:40.727310076 (524h17m26.189402549s) [upload=461218]: Waiting for error group managing segments... Oct 15 09:48:41.004514306 (524h17m26.466606789s) [noise=true , address=145.79.227.7:4294 , piece_id=H4ZNVIXORNO4MTXOVYDQZI4IJE4VXHZWC6JI35YKOR3UESSXJNTQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.006116557 (524h17m26.46820904s) [noise=true , address=145.79.227.5:4222 , piece_id=C2JSPN2QVSGSXG6ZW3ZMOFX5YFNN65XY7E62N5OEHOOMK3NSO4KQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.012963184 (524h17m26.475055667s) [noise=true , address=145.79.227.16:4910 , piece_id=BIHEJDH7RPI4UCPUNV453QHSEVCLA54PSNNQB2II37XZDRXFZNWQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.012994684 (524h17m26.475087167s) [noise=true , address=145.79.227.14:4840 , piece_id=B4ZVPRMVH7Y3TCWCLPO7DWRVMJL7VSI73CIT4PEHCMTIEURQKB2Q, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.016649956 (524h17m26.478742439s) [noise=true , address=145.79.227.11:4730 , piece_id=VO2DSM5LT43QAFPWOY3W3PVEJ44GZJWTCWG5QRQQFB363DFVFZDA, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.019248864 (524h17m26.481341347s) [noise=true , address=145.79.227.17:4969 , piece_id=R5HIP33QW5HLICARFWDUOW7I75YTDHZTJCIML5X52VKHLPL3J4YQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.075991585 (524h17m26.538084068s) [noise=true , address=145.79.227.4:4187 , piece_id=TQVPL6ZA6LHKAJVL6XWSYASZD3MCKKV3553CRFD72XWTVWY62STQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.118655146 (524h17m26.580747619s) [noise=true , address=145.79.227.14:4838 , piece_id=6K6A4CNEHGB3X6DEBLF5KHLUT4GE6BQBW6IE3RHBML2NWPN255AA, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.285934795 (524h17m26.748027278s) [noise=true , address=77.90.31.11:4555 , piece_id=U5RYZSD643KSUI4XLL26EAN6FQRPMG62GQMMCQRPEEG2XKSNBIZQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.744525465 (524h17m27.206617948s) [noise=true , address=45.91.116.9:4262 , piece_id=X53JZGRLZLX53UZLLCXXUQCFSQD2FO76X74OM45G26HNDU55UXXQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:42.218961989 (524h17m27.681054462s) [noise=true , address=45.91.116.11:4117 , piece_id=VH7ZJKSZ2NUXQJZE2IKNWP5PAL3TN2NNHD3ATZUQHGKKYMSSM3LQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:42.313638675 (524h17m27.775731158s) [noise=true , address=45.91.116.14:4189 , piece_id=UMQRQYPZ7SWUCO2N7JDCEVI5U5P36FDL3SUM6HIYJVWVOZTWNXQQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:42.434283559 (524h17m27.896376053s) [noise=true , address=45.91.116.8:4045 , piece_id=EDXBNLKKRZMT6MKK2KOIZM7DDKA3IHB6UMCXX4YKZF7JY2T3S5CQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:42.434321832 (524h17m27.896414305s) [seg_pos={0 0}, upload=461218]: Segment reached optimal threshold of 13 pieces. Oct 15 09:48:42.434605041 (524h17m27.896697534s) [noise=true , address=45.91.116.5:4299 , piece_id=4ARMXXE5LBRE2OZQEIITELKPTW4TISZWTD6XZGI4UJV7UCLUXAEQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX): protocol: expected piece hash; context canceled; EOF Oct 15 09:48:42.434666138 (524h17m27.896758621s) [noise=true , address=45.91.116.5:4299 , piece_id=4ARMXXE5LBRE2OZQEIITELKPTW4TISZWTD6XZGI4UJV7UCLUXAEQ, seg_pos={0 0}, upload=461218]: Long tail cancellation due to optimal threshold Oct 15 09:48:42.434692849 (524h17m27.896785432s) [noise=true , address=45.91.116.12:4226 , piece_id=DRXRGO7OXEKVS5ACL6FC7RKLGBBD6KNI7X6E5DM5D5VV5PEKN7FQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1): protocol: expected piece hash; context canceled; EOF Oct 15 09:48:42.434706184 (524h17m27.896798657s) [noise=true , address=45.91.116.12:4226 , piece_id=DRXRGO7OXEKVS5ACL6FC7RKLGBBD6KNI7X6E5DM5D5VV5PEKN7FQ, seg_pos={0 0}, upload=461218]: Long tail cancellation due to optimal threshold Oct 15 09:48:42.434718318 (524h17m27.896810911s) [noise=true , address=45.91.116.4:4940 , piece_id=3MCSUDYN73NIVJTGKPOVY6N3MALB3GBXI7I5NAX4HXPDHVMQ52BA, seg_pos={0 0}, upload=461218]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe): protocol: expected piece hash; context canceled; EOF Oct 15 09:48:42.434728447 (524h17m27.89682093s) [noise=true , address=45.91.116.4:4940 , piece_id=3MCSUDYN73NIVJTGKPOVY6N3MALB3GBXI7I5NAX4HXPDHVMQ52BA, seg_pos={0 0}, upload=461218]: Long tail cancellation due to optimal threshold Oct 15 09:48:42.434760458 (524h17m27.896852931s) [seg_pos={0 0}, upload=461218]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 15 09:48:42.434866900 (524h17m27.896959384s) [upload=461218]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 15 09:48:42.434873884 (524h17m27.896966367s) [upload=461218]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 15 09:48:42.760633614 (524h17m28.222726087s) [upload=461218]: Done uploading segments... Oct 15 09:48:42.760655435 (524h17m28.222747918s) [upload=461218]: Upload finished. err: <nil> Duration: 5.932115753s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [............X...X...........................................] duration: 6.241156609s FAIL (2025-10-15 10:21:13 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, Piece ID: VH7ZJKSZ2NUXQJZE2IKNWP5PAL3TN2NNHD3ATZUQHGKKYMSSM3LQ) context deadline exceeded; piecestore: (Node ID: 1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N, Piece ID: UMQRQYPZ7SWUCO2N7JDCEVI5U5P36FDL3SUM6HIYJVWVOZTWNXQQ) context deadline exceeded ======LOGS====== Oct 15 09:48:38.801058027 (524h17m24.26315051s) [upload=461218]: Starting upload Oct 15 09:48:38.801138119 (524h17m24.263230592s) [upload=461218]: Done starting upload Oct 15 09:48:38.801288545 (524h17m24.263381028s) [upload=461218]: Uploading segments... Oct 15 09:48:38.801442509 (524h17m24.263535002s) [upload=461218]: Got next segment. Inline: false Oct 15 09:48:38.801486663 (524h17m24.263579136s) [upload=461218]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 15 09:48:38.801493245 (524h17m24.263585728s) [upload=461218]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 15 09:48:39.066131918 (524h17m24.528224401s) [seg_pos={0 0}, upload=461218]: Begin upload segment... Oct 15 09:48:39.066205549 (524h17m24.528298022s) [seg_pos={0 0}, upload=461218]: Done begin upload segment. Oct 15 09:48:39.066243090 (524h17m24.528335573s) [noise=true , address=45.91.116.5:4299 , piece_id=4ARMXXE5LBRE2OZQEIITELKPTW4TISZWTD6XZGI4UJV7UCLUXAEQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.066254431 (524h17m24.528346915s) [noise=true , address=45.91.116.11:4117 , piece_id=VH7ZJKSZ2NUXQJZE2IKNWP5PAL3TN2NNHD3ATZUQHGKKYMSSM3LQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.066874873 (524h17m24.528967356s) [noise=true , address=145.79.227.7:4294 , piece_id=H4ZNVIXORNO4MTXOVYDQZI4IJE4VXHZWC6JI35YKOR3UESSXJNTQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.067301676 (524h17m24.529394159s) [noise=true , address=45.91.116.9:4262 , piece_id=X53JZGRLZLX53UZLLCXXUQCFSQD2FO76X74OM45G26HNDU55UXXQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.067705735 (524h17m24.529798218s) [noise=true , address=145.79.227.4:4187 , piece_id=TQVPL6ZA6LHKAJVL6XWSYASZD3MCKKV3553CRFD72XWTVWY62STQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.069183369 (524h17m24.531275862s) [noise=true , address=145.79.227.5:4222 , piece_id=C2JSPN2QVSGSXG6ZW3ZMOFX5YFNN65XY7E62N5OEHOOMK3NSO4KQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.070534782 (524h17m24.532627265s) [noise=true , address=77.90.31.11:4555 , piece_id=U5RYZSD643KSUI4XLL26EAN6FQRPMG62GQMMCQRPEEG2XKSNBIZQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.072356982 (524h17m24.534449466s) [noise=true , address=145.79.227.11:4730 , piece_id=VO2DSM5LT43QAFPWOY3W3PVEJ44GZJWTCWG5QRQQFB363DFVFZDA, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.073961167 (524h17m24.53605364s) [noise=true , address=45.91.116.4:4940 , piece_id=3MCSUDYN73NIVJTGKPOVY6N3MALB3GBXI7I5NAX4HXPDHVMQ52BA, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.075716960 (524h17m24.537809433s) [noise=true , address=145.79.227.16:4910 , piece_id=BIHEJDH7RPI4UCPUNV453QHSEVCLA54PSNNQB2II37XZDRXFZNWQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.079266419 (524h17m24.541358902s) [noise=true , address=145.79.227.14:4838 , piece_id=6K6A4CNEHGB3X6DEBLF5KHLUT4GE6BQBW6IE3RHBML2NWPN255AA, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.080943032 (524h17m24.543035505s) [noise=true , address=45.91.116.12:4226 , piece_id=DRXRGO7OXEKVS5ACL6FC7RKLGBBD6KNI7X6E5DM5D5VV5PEKN7FQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.081340479 (524h17m24.543432962s) [noise=true , address=45.91.116.8:4045 , piece_id=EDXBNLKKRZMT6MKK2KOIZM7DDKA3IHB6UMCXX4YKZF7JY2T3S5CQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.083616223 (524h17m24.545708706s) [noise=true , address=145.79.227.14:4840 , piece_id=B4ZVPRMVH7Y3TCWCLPO7DWRVMJL7VSI73CIT4PEHCMTIEURQKB2Q, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.083633946 (524h17m24.545726429s) [noise=true , address=45.91.116.14:4189 , piece_id=UMQRQYPZ7SWUCO2N7JDCEVI5U5P36FDL3SUM6HIYJVWVOZTWNXQQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.085493247 (524h17m24.54758573s) [noise=true , address=145.79.227.17:4969 , piece_id=R5HIP33QW5HLICARFWDUOW7I75YTDHZTJCIML5X52VKHLPL3J4YQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:40.727263388 (524h17m26.189355871s) [upload=461218]: Next returned nil segment Oct 15 09:48:40.727310076 (524h17m26.189402549s) [upload=461218]: Waiting for error group managing segments... Oct 15 09:48:41.004514306 (524h17m26.466606789s) [noise=true , address=145.79.227.7:4294 , piece_id=H4ZNVIXORNO4MTXOVYDQZI4IJE4VXHZWC6JI35YKOR3UESSXJNTQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.006116557 (524h17m26.46820904s) [noise=true , address=145.79.227.5:4222 , piece_id=C2JSPN2QVSGSXG6ZW3ZMOFX5YFNN65XY7E62N5OEHOOMK3NSO4KQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.012963184 (524h17m26.475055667s) [noise=true , address=145.79.227.16:4910 , piece_id=BIHEJDH7RPI4UCPUNV453QHSEVCLA54PSNNQB2II37XZDRXFZNWQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.012994684 (524h17m26.475087167s) [noise=true , address=145.79.227.14:4840 , piece_id=B4ZVPRMVH7Y3TCWCLPO7DWRVMJL7VSI73CIT4PEHCMTIEURQKB2Q, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.016649956 (524h17m26.478742439s) [noise=true , address=145.79.227.11:4730 , piece_id=VO2DSM5LT43QAFPWOY3W3PVEJ44GZJWTCWG5QRQQFB363DFVFZDA, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.019248864 (524h17m26.481341347s) [noise=true , address=145.79.227.17:4969 , piece_id=R5HIP33QW5HLICARFWDUOW7I75YTDHZTJCIML5X52VKHLPL3J4YQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.075991585 (524h17m26.538084068s) [noise=true , address=145.79.227.4:4187 , piece_id=TQVPL6ZA6LHKAJVL6XWSYASZD3MCKKV3553CRFD72XWTVWY62STQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.118655146 (524h17m26.580747619s) [noise=true , address=145.79.227.14:4838 , piece_id=6K6A4CNEHGB3X6DEBLF5KHLUT4GE6BQBW6IE3RHBML2NWPN255AA, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.285934795 (524h17m26.748027278s) [noise=true , address=77.90.31.11:4555 , piece_id=U5RYZSD643KSUI4XLL26EAN6FQRPMG62GQMMCQRPEEG2XKSNBIZQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.744525465 (524h17m27.206617948s) [noise=true , address=45.91.116.9:4262 , piece_id=X53JZGRLZLX53UZLLCXXUQCFSQD2FO76X74OM45G26HNDU55UXXQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:42.218961989 (524h17m27.681054462s) [noise=true , address=45.91.116.11:4117 , piece_id=VH7ZJKSZ2NUXQJZE2IKNWP5PAL3TN2NNHD3ATZUQHGKKYMSSM3LQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:42.313638675 (524h17m27.775731158s) [noise=true , address=45.91.116.14:4189 , piece_id=UMQRQYPZ7SWUCO2N7JDCEVI5U5P36FDL3SUM6HIYJVWVOZTWNXQQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:42.434283559 (524h17m27.896376053s) [noise=true , address=45.91.116.8:4045 , piece_id=EDXBNLKKRZMT6MKK2KOIZM7DDKA3IHB6UMCXX4YKZF7JY2T3S5CQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:42.434321832 (524h17m27.896414305s) [seg_pos={0 0}, upload=461218]: Segment reached optimal threshold of 13 pieces. Oct 15 09:48:42.434605041 (524h17m27.896697534s) [noise=true , address=45.91.116.5:4299 , piece_id=4ARMXXE5LBRE2OZQEIITELKPTW4TISZWTD6XZGI4UJV7UCLUXAEQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX): protocol: expected piece hash; context canceled; EOF Oct 15 09:48:42.434666138 (524h17m27.896758621s) [noise=true , address=45.91.116.5:4299 , piece_id=4ARMXXE5LBRE2OZQEIITELKPTW4TISZWTD6XZGI4UJV7UCLUXAEQ, seg_pos={0 0}, upload=461218]: Long tail cancellation due to optimal threshold Oct 15 09:48:42.434692849 (524h17m27.896785432s) [noise=true , address=45.91.116.12:4226 , piece_id=DRXRGO7OXEKVS5ACL6FC7RKLGBBD6KNI7X6E5DM5D5VV5PEKN7FQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1): protocol: expected piece hash; context canceled; EOF Oct 15 09:48:42.434706184 (524h17m27.896798657s) [noise=true , address=45.91.116.12:4226 , piece_id=DRXRGO7OXEKVS5ACL6FC7RKLGBBD6KNI7X6E5DM5D5VV5PEKN7FQ, seg_pos={0 0}, upload=461218]: Long tail cancellation due to optimal threshold Oct 15 09:48:42.434718318 (524h17m27.896810911s) [noise=true , address=45.91.116.4:4940 , piece_id=3MCSUDYN73NIVJTGKPOVY6N3MALB3GBXI7I5NAX4HXPDHVMQ52BA, seg_pos={0 0}, upload=461218]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe): protocol: expected piece hash; context canceled; EOF Oct 15 09:48:42.434728447 (524h17m27.89682093s) [noise=true , address=45.91.116.4:4940 , piece_id=3MCSUDYN73NIVJTGKPOVY6N3MALB3GBXI7I5NAX4HXPDHVMQ52BA, seg_pos={0 0}, upload=461218]: Long tail cancellation due to optimal threshold Oct 15 09:48:42.434760458 (524h17m27.896852931s) [seg_pos={0 0}, upload=461218]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 15 09:48:42.434866900 (524h17m27.896959384s) [upload=461218]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 15 09:48:42.434873884 (524h17m27.896966367s) [upload=461218]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 15 09:48:42.760633614 (524h17m28.222726087s) [upload=461218]: Done uploading segments... Oct 15 09:48:42.760655435 (524h17m28.222747918s) [upload=461218]: Upload finished. err: <nil> Duration: 6.241182638s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.............X...X..........................................] duration: 6.12004862s FAIL (2025-10-15 10:20:21 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, Piece ID: VH7ZJKSZ2NUXQJZE2IKNWP5PAL3TN2NNHD3ATZUQHGKKYMSSM3LQ) context deadline exceeded; piecestore: (Node ID: 1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N, Piece ID: UMQRQYPZ7SWUCO2N7JDCEVI5U5P36FDL3SUM6HIYJVWVOZTWNXQQ) context deadline exceeded ======LOGS====== Oct 15 09:48:38.801058027 (524h17m24.26315051s) [upload=461218]: Starting upload Oct 15 09:48:38.801138119 (524h17m24.263230592s) [upload=461218]: Done starting upload Oct 15 09:48:38.801288545 (524h17m24.263381028s) [upload=461218]: Uploading segments... Oct 15 09:48:38.801442509 (524h17m24.263535002s) [upload=461218]: Got next segment. Inline: false Oct 15 09:48:38.801486663 (524h17m24.263579136s) [upload=461218]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 15 09:48:38.801493245 (524h17m24.263585728s) [upload=461218]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 15 09:48:39.066131918 (524h17m24.528224401s) [seg_pos={0 0}, upload=461218]: Begin upload segment... Oct 15 09:48:39.066205549 (524h17m24.528298022s) [seg_pos={0 0}, upload=461218]: Done begin upload segment. Oct 15 09:48:39.066243090 (524h17m24.528335573s) [noise=true , address=45.91.116.5:4299 , piece_id=4ARMXXE5LBRE2OZQEIITELKPTW4TISZWTD6XZGI4UJV7UCLUXAEQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.066254431 (524h17m24.528346915s) [noise=true , address=45.91.116.11:4117 , piece_id=VH7ZJKSZ2NUXQJZE2IKNWP5PAL3TN2NNHD3ATZUQHGKKYMSSM3LQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.066874873 (524h17m24.528967356s) [noise=true , address=145.79.227.7:4294 , piece_id=H4ZNVIXORNO4MTXOVYDQZI4IJE4VXHZWC6JI35YKOR3UESSXJNTQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.067301676 (524h17m24.529394159s) [noise=true , address=45.91.116.9:4262 , piece_id=X53JZGRLZLX53UZLLCXXUQCFSQD2FO76X74OM45G26HNDU55UXXQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.067705735 (524h17m24.529798218s) [noise=true , address=145.79.227.4:4187 , piece_id=TQVPL6ZA6LHKAJVL6XWSYASZD3MCKKV3553CRFD72XWTVWY62STQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.069183369 (524h17m24.531275862s) [noise=true , address=145.79.227.5:4222 , piece_id=C2JSPN2QVSGSXG6ZW3ZMOFX5YFNN65XY7E62N5OEHOOMK3NSO4KQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.070534782 (524h17m24.532627265s) [noise=true , address=77.90.31.11:4555 , piece_id=U5RYZSD643KSUI4XLL26EAN6FQRPMG62GQMMCQRPEEG2XKSNBIZQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.072356982 (524h17m24.534449466s) [noise=true , address=145.79.227.11:4730 , piece_id=VO2DSM5LT43QAFPWOY3W3PVEJ44GZJWTCWG5QRQQFB363DFVFZDA, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.073961167 (524h17m24.53605364s) [noise=true , address=45.91.116.4:4940 , piece_id=3MCSUDYN73NIVJTGKPOVY6N3MALB3GBXI7I5NAX4HXPDHVMQ52BA, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.075716960 (524h17m24.537809433s) [noise=true , address=145.79.227.16:4910 , piece_id=BIHEJDH7RPI4UCPUNV453QHSEVCLA54PSNNQB2II37XZDRXFZNWQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.079266419 (524h17m24.541358902s) [noise=true , address=145.79.227.14:4838 , piece_id=6K6A4CNEHGB3X6DEBLF5KHLUT4GE6BQBW6IE3RHBML2NWPN255AA, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.080943032 (524h17m24.543035505s) [noise=true , address=45.91.116.12:4226 , piece_id=DRXRGO7OXEKVS5ACL6FC7RKLGBBD6KNI7X6E5DM5D5VV5PEKN7FQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.081340479 (524h17m24.543432962s) [noise=true , address=45.91.116.8:4045 , piece_id=EDXBNLKKRZMT6MKK2KOIZM7DDKA3IHB6UMCXX4YKZF7JY2T3S5CQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.083616223 (524h17m24.545708706s) [noise=true , address=145.79.227.14:4840 , piece_id=B4ZVPRMVH7Y3TCWCLPO7DWRVMJL7VSI73CIT4PEHCMTIEURQKB2Q, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.083633946 (524h17m24.545726429s) [noise=true , address=45.91.116.14:4189 , piece_id=UMQRQYPZ7SWUCO2N7JDCEVI5U5P36FDL3SUM6HIYJVWVOZTWNXQQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.085493247 (524h17m24.54758573s) [noise=true , address=145.79.227.17:4969 , piece_id=R5HIP33QW5HLICARFWDUOW7I75YTDHZTJCIML5X52VKHLPL3J4YQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:40.727263388 (524h17m26.189355871s) [upload=461218]: Next returned nil segment Oct 15 09:48:40.727310076 (524h17m26.189402549s) [upload=461218]: Waiting for error group managing segments... Oct 15 09:48:41.004514306 (524h17m26.466606789s) [noise=true , address=145.79.227.7:4294 , piece_id=H4ZNVIXORNO4MTXOVYDQZI4IJE4VXHZWC6JI35YKOR3UESSXJNTQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.006116557 (524h17m26.46820904s) [noise=true , address=145.79.227.5:4222 , piece_id=C2JSPN2QVSGSXG6ZW3ZMOFX5YFNN65XY7E62N5OEHOOMK3NSO4KQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.012963184 (524h17m26.475055667s) [noise=true , address=145.79.227.16:4910 , piece_id=BIHEJDH7RPI4UCPUNV453QHSEVCLA54PSNNQB2II37XZDRXFZNWQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.012994684 (524h17m26.475087167s) [noise=true , address=145.79.227.14:4840 , piece_id=B4ZVPRMVH7Y3TCWCLPO7DWRVMJL7VSI73CIT4PEHCMTIEURQKB2Q, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.016649956 (524h17m26.478742439s) [noise=true , address=145.79.227.11:4730 , piece_id=VO2DSM5LT43QAFPWOY3W3PVEJ44GZJWTCWG5QRQQFB363DFVFZDA, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.019248864 (524h17m26.481341347s) [noise=true , address=145.79.227.17:4969 , piece_id=R5HIP33QW5HLICARFWDUOW7I75YTDHZTJCIML5X52VKHLPL3J4YQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.075991585 (524h17m26.538084068s) [noise=true , address=145.79.227.4:4187 , piece_id=TQVPL6ZA6LHKAJVL6XWSYASZD3MCKKV3553CRFD72XWTVWY62STQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.118655146 (524h17m26.580747619s) [noise=true , address=145.79.227.14:4838 , piece_id=6K6A4CNEHGB3X6DEBLF5KHLUT4GE6BQBW6IE3RHBML2NWPN255AA, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.285934795 (524h17m26.748027278s) [noise=true , address=77.90.31.11:4555 , piece_id=U5RYZSD643KSUI4XLL26EAN6FQRPMG62GQMMCQRPEEG2XKSNBIZQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.744525465 (524h17m27.206617948s) [noise=true , address=45.91.116.9:4262 , piece_id=X53JZGRLZLX53UZLLCXXUQCFSQD2FO76X74OM45G26HNDU55UXXQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:42.218961989 (524h17m27.681054462s) [noise=true , address=45.91.116.11:4117 , piece_id=VH7ZJKSZ2NUXQJZE2IKNWP5PAL3TN2NNHD3ATZUQHGKKYMSSM3LQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:42.313638675 (524h17m27.775731158s) [noise=true , address=45.91.116.14:4189 , piece_id=UMQRQYPZ7SWUCO2N7JDCEVI5U5P36FDL3SUM6HIYJVWVOZTWNXQQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:42.434283559 (524h17m27.896376053s) [noise=true , address=45.91.116.8:4045 , piece_id=EDXBNLKKRZMT6MKK2KOIZM7DDKA3IHB6UMCXX4YKZF7JY2T3S5CQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:42.434321832 (524h17m27.896414305s) [seg_pos={0 0}, upload=461218]: Segment reached optimal threshold of 13 pieces. Oct 15 09:48:42.434605041 (524h17m27.896697534s) [noise=true , address=45.91.116.5:4299 , piece_id=4ARMXXE5LBRE2OZQEIITELKPTW4TISZWTD6XZGI4UJV7UCLUXAEQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX): protocol: expected piece hash; context canceled; EOF Oct 15 09:48:42.434666138 (524h17m27.896758621s) [noise=true , address=45.91.116.5:4299 , piece_id=4ARMXXE5LBRE2OZQEIITELKPTW4TISZWTD6XZGI4UJV7UCLUXAEQ, seg_pos={0 0}, upload=461218]: Long tail cancellation due to optimal threshold Oct 15 09:48:42.434692849 (524h17m27.896785432s) [noise=true , address=45.91.116.12:4226 , piece_id=DRXRGO7OXEKVS5ACL6FC7RKLGBBD6KNI7X6E5DM5D5VV5PEKN7FQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1): protocol: expected piece hash; context canceled; EOF Oct 15 09:48:42.434706184 (524h17m27.896798657s) [noise=true , address=45.91.116.12:4226 , piece_id=DRXRGO7OXEKVS5ACL6FC7RKLGBBD6KNI7X6E5DM5D5VV5PEKN7FQ, seg_pos={0 0}, upload=461218]: Long tail cancellation due to optimal threshold Oct 15 09:48:42.434718318 (524h17m27.896810911s) [noise=true , address=45.91.116.4:4940 , piece_id=3MCSUDYN73NIVJTGKPOVY6N3MALB3GBXI7I5NAX4HXPDHVMQ52BA, seg_pos={0 0}, upload=461218]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe): protocol: expected piece hash; context canceled; EOF Oct 15 09:48:42.434728447 (524h17m27.89682093s) [noise=true , address=45.91.116.4:4940 , piece_id=3MCSUDYN73NIVJTGKPOVY6N3MALB3GBXI7I5NAX4HXPDHVMQ52BA, seg_pos={0 0}, upload=461218]: Long tail cancellation due to optimal threshold Oct 15 09:48:42.434760458 (524h17m27.896852931s) [seg_pos={0 0}, upload=461218]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 15 09:48:42.434866900 (524h17m27.896959384s) [upload=461218]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 15 09:48:42.434873884 (524h17m27.896966367s) [upload=461218]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 15 09:48:42.760633614 (524h17m28.222726087s) [upload=461218]: Done uploading segments... Oct 15 09:48:42.760655435 (524h17m28.222747918s) [upload=461218]: Upload finished. err: <nil> Duration: 6.120113664s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..............X...X.........................................] duration: 6.765888922s FAIL (2025-10-15 10:19:12 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, Piece ID: VH7ZJKSZ2NUXQJZE2IKNWP5PAL3TN2NNHD3ATZUQHGKKYMSSM3LQ) context deadline exceeded; piecestore: (Node ID: 1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N, Piece ID: UMQRQYPZ7SWUCO2N7JDCEVI5U5P36FDL3SUM6HIYJVWVOZTWNXQQ) context deadline exceeded ======LOGS====== Oct 15 09:48:38.801058027 (524h17m24.26315051s) [upload=461218]: Starting upload Oct 15 09:48:38.801138119 (524h17m24.263230592s) [upload=461218]: Done starting upload Oct 15 09:48:38.801288545 (524h17m24.263381028s) [upload=461218]: Uploading segments... Oct 15 09:48:38.801442509 (524h17m24.263535002s) [upload=461218]: Got next segment. Inline: false Oct 15 09:48:38.801486663 (524h17m24.263579136s) [upload=461218]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 15 09:48:38.801493245 (524h17m24.263585728s) [upload=461218]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 15 09:48:39.066131918 (524h17m24.528224401s) [seg_pos={0 0}, upload=461218]: Begin upload segment... Oct 15 09:48:39.066205549 (524h17m24.528298022s) [seg_pos={0 0}, upload=461218]: Done begin upload segment. Oct 15 09:48:39.066243090 (524h17m24.528335573s) [noise=true , address=45.91.116.5:4299 , piece_id=4ARMXXE5LBRE2OZQEIITELKPTW4TISZWTD6XZGI4UJV7UCLUXAEQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.066254431 (524h17m24.528346915s) [noise=true , address=45.91.116.11:4117 , piece_id=VH7ZJKSZ2NUXQJZE2IKNWP5PAL3TN2NNHD3ATZUQHGKKYMSSM3LQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.066874873 (524h17m24.528967356s) [noise=true , address=145.79.227.7:4294 , piece_id=H4ZNVIXORNO4MTXOVYDQZI4IJE4VXHZWC6JI35YKOR3UESSXJNTQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.067301676 (524h17m24.529394159s) [noise=true , address=45.91.116.9:4262 , piece_id=X53JZGRLZLX53UZLLCXXUQCFSQD2FO76X74OM45G26HNDU55UXXQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.067705735 (524h17m24.529798218s) [noise=true , address=145.79.227.4:4187 , piece_id=TQVPL6ZA6LHKAJVL6XWSYASZD3MCKKV3553CRFD72XWTVWY62STQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.069183369 (524h17m24.531275862s) [noise=true , address=145.79.227.5:4222 , piece_id=C2JSPN2QVSGSXG6ZW3ZMOFX5YFNN65XY7E62N5OEHOOMK3NSO4KQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.070534782 (524h17m24.532627265s) [noise=true , address=77.90.31.11:4555 , piece_id=U5RYZSD643KSUI4XLL26EAN6FQRPMG62GQMMCQRPEEG2XKSNBIZQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.072356982 (524h17m24.534449466s) [noise=true , address=145.79.227.11:4730 , piece_id=VO2DSM5LT43QAFPWOY3W3PVEJ44GZJWTCWG5QRQQFB363DFVFZDA, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.073961167 (524h17m24.53605364s) [noise=true , address=45.91.116.4:4940 , piece_id=3MCSUDYN73NIVJTGKPOVY6N3MALB3GBXI7I5NAX4HXPDHVMQ52BA, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.075716960 (524h17m24.537809433s) [noise=true , address=145.79.227.16:4910 , piece_id=BIHEJDH7RPI4UCPUNV453QHSEVCLA54PSNNQB2II37XZDRXFZNWQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.079266419 (524h17m24.541358902s) [noise=true , address=145.79.227.14:4838 , piece_id=6K6A4CNEHGB3X6DEBLF5KHLUT4GE6BQBW6IE3RHBML2NWPN255AA, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.080943032 (524h17m24.543035505s) [noise=true , address=45.91.116.12:4226 , piece_id=DRXRGO7OXEKVS5ACL6FC7RKLGBBD6KNI7X6E5DM5D5VV5PEKN7FQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.081340479 (524h17m24.543432962s) [noise=true , address=45.91.116.8:4045 , piece_id=EDXBNLKKRZMT6MKK2KOIZM7DDKA3IHB6UMCXX4YKZF7JY2T3S5CQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.083616223 (524h17m24.545708706s) [noise=true , address=145.79.227.14:4840 , piece_id=B4ZVPRMVH7Y3TCWCLPO7DWRVMJL7VSI73CIT4PEHCMTIEURQKB2Q, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.083633946 (524h17m24.545726429s) [noise=true , address=45.91.116.14:4189 , piece_id=UMQRQYPZ7SWUCO2N7JDCEVI5U5P36FDL3SUM6HIYJVWVOZTWNXQQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.085493247 (524h17m24.54758573s) [noise=true , address=145.79.227.17:4969 , piece_id=R5HIP33QW5HLICARFWDUOW7I75YTDHZTJCIML5X52VKHLPL3J4YQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:40.727263388 (524h17m26.189355871s) [upload=461218]: Next returned nil segment Oct 15 09:48:40.727310076 (524h17m26.189402549s) [upload=461218]: Waiting for error group managing segments... Oct 15 09:48:41.004514306 (524h17m26.466606789s) [noise=true , address=145.79.227.7:4294 , piece_id=H4ZNVIXORNO4MTXOVYDQZI4IJE4VXHZWC6JI35YKOR3UESSXJNTQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.006116557 (524h17m26.46820904s) [noise=true , address=145.79.227.5:4222 , piece_id=C2JSPN2QVSGSXG6ZW3ZMOFX5YFNN65XY7E62N5OEHOOMK3NSO4KQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.012963184 (524h17m26.475055667s) [noise=true , address=145.79.227.16:4910 , piece_id=BIHEJDH7RPI4UCPUNV453QHSEVCLA54PSNNQB2II37XZDRXFZNWQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.012994684 (524h17m26.475087167s) [noise=true , address=145.79.227.14:4840 , piece_id=B4ZVPRMVH7Y3TCWCLPO7DWRVMJL7VSI73CIT4PEHCMTIEURQKB2Q, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.016649956 (524h17m26.478742439s) [noise=true , address=145.79.227.11:4730 , piece_id=VO2DSM5LT43QAFPWOY3W3PVEJ44GZJWTCWG5QRQQFB363DFVFZDA, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.019248864 (524h17m26.481341347s) [noise=true , address=145.79.227.17:4969 , piece_id=R5HIP33QW5HLICARFWDUOW7I75YTDHZTJCIML5X52VKHLPL3J4YQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.075991585 (524h17m26.538084068s) [noise=true , address=145.79.227.4:4187 , piece_id=TQVPL6ZA6LHKAJVL6XWSYASZD3MCKKV3553CRFD72XWTVWY62STQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.118655146 (524h17m26.580747619s) [noise=true , address=145.79.227.14:4838 , piece_id=6K6A4CNEHGB3X6DEBLF5KHLUT4GE6BQBW6IE3RHBML2NWPN255AA, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.285934795 (524h17m26.748027278s) [noise=true , address=77.90.31.11:4555 , piece_id=U5RYZSD643KSUI4XLL26EAN6FQRPMG62GQMMCQRPEEG2XKSNBIZQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.744525465 (524h17m27.206617948s) [noise=true , address=45.91.116.9:4262 , piece_id=X53JZGRLZLX53UZLLCXXUQCFSQD2FO76X74OM45G26HNDU55UXXQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:42.218961989 (524h17m27.681054462s) [noise=true , address=45.91.116.11:4117 , piece_id=VH7ZJKSZ2NUXQJZE2IKNWP5PAL3TN2NNHD3ATZUQHGKKYMSSM3LQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:42.313638675 (524h17m27.775731158s) [noise=true , address=45.91.116.14:4189 , piece_id=UMQRQYPZ7SWUCO2N7JDCEVI5U5P36FDL3SUM6HIYJVWVOZTWNXQQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:42.434283559 (524h17m27.896376053s) [noise=true , address=45.91.116.8:4045 , piece_id=EDXBNLKKRZMT6MKK2KOIZM7DDKA3IHB6UMCXX4YKZF7JY2T3S5CQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:42.434321832 (524h17m27.896414305s) [seg_pos={0 0}, upload=461218]: Segment reached optimal threshold of 13 pieces. Oct 15 09:48:42.434605041 (524h17m27.896697534s) [noise=true , address=45.91.116.5:4299 , piece_id=4ARMXXE5LBRE2OZQEIITELKPTW4TISZWTD6XZGI4UJV7UCLUXAEQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX): protocol: expected piece hash; context canceled; EOF Oct 15 09:48:42.434666138 (524h17m27.896758621s) [noise=true , address=45.91.116.5:4299 , piece_id=4ARMXXE5LBRE2OZQEIITELKPTW4TISZWTD6XZGI4UJV7UCLUXAEQ, seg_pos={0 0}, upload=461218]: Long tail cancellation due to optimal threshold Oct 15 09:48:42.434692849 (524h17m27.896785432s) [noise=true , address=45.91.116.12:4226 , piece_id=DRXRGO7OXEKVS5ACL6FC7RKLGBBD6KNI7X6E5DM5D5VV5PEKN7FQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1): protocol: expected piece hash; context canceled; EOF Oct 15 09:48:42.434706184 (524h17m27.896798657s) [noise=true , address=45.91.116.12:4226 , piece_id=DRXRGO7OXEKVS5ACL6FC7RKLGBBD6KNI7X6E5DM5D5VV5PEKN7FQ, seg_pos={0 0}, upload=461218]: Long tail cancellation due to optimal threshold Oct 15 09:48:42.434718318 (524h17m27.896810911s) [noise=true , address=45.91.116.4:4940 , piece_id=3MCSUDYN73NIVJTGKPOVY6N3MALB3GBXI7I5NAX4HXPDHVMQ52BA, seg_pos={0 0}, upload=461218]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe): protocol: expected piece hash; context canceled; EOF Oct 15 09:48:42.434728447 (524h17m27.89682093s) [noise=true , address=45.91.116.4:4940 , piece_id=3MCSUDYN73NIVJTGKPOVY6N3MALB3GBXI7I5NAX4HXPDHVMQ52BA, seg_pos={0 0}, upload=461218]: Long tail cancellation due to optimal threshold Oct 15 09:48:42.434760458 (524h17m27.896852931s) [seg_pos={0 0}, upload=461218]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 15 09:48:42.434866900 (524h17m27.896959384s) [upload=461218]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 15 09:48:42.434873884 (524h17m27.896966367s) [upload=461218]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 15 09:48:42.760633614 (524h17m28.222726087s) [upload=461218]: Done uploading segments... Oct 15 09:48:42.760655435 (524h17m28.222747918s) [upload=461218]: Upload finished. err: <nil> Duration: 6.765929038s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...............X...X........................................] duration: 6.252904578s FAIL (2025-10-15 10:18:47 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, Piece ID: VH7ZJKSZ2NUXQJZE2IKNWP5PAL3TN2NNHD3ATZUQHGKKYMSSM3LQ) context deadline exceeded; piecestore: (Node ID: 1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N, Piece ID: UMQRQYPZ7SWUCO2N7JDCEVI5U5P36FDL3SUM6HIYJVWVOZTWNXQQ) context deadline exceeded ======LOGS====== Oct 15 09:48:38.801058027 (524h17m24.26315051s) [upload=461218]: Starting upload Oct 15 09:48:38.801138119 (524h17m24.263230592s) [upload=461218]: Done starting upload Oct 15 09:48:38.801288545 (524h17m24.263381028s) [upload=461218]: Uploading segments... Oct 15 09:48:38.801442509 (524h17m24.263535002s) [upload=461218]: Got next segment. Inline: false Oct 15 09:48:38.801486663 (524h17m24.263579136s) [upload=461218]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 15 09:48:38.801493245 (524h17m24.263585728s) [upload=461218]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 15 09:48:39.066131918 (524h17m24.528224401s) [seg_pos={0 0}, upload=461218]: Begin upload segment... Oct 15 09:48:39.066205549 (524h17m24.528298022s) [seg_pos={0 0}, upload=461218]: Done begin upload segment. Oct 15 09:48:39.066243090 (524h17m24.528335573s) [noise=true , address=45.91.116.5:4299 , piece_id=4ARMXXE5LBRE2OZQEIITELKPTW4TISZWTD6XZGI4UJV7UCLUXAEQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.066254431 (524h17m24.528346915s) [noise=true , address=45.91.116.11:4117 , piece_id=VH7ZJKSZ2NUXQJZE2IKNWP5PAL3TN2NNHD3ATZUQHGKKYMSSM3LQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.066874873 (524h17m24.528967356s) [noise=true , address=145.79.227.7:4294 , piece_id=H4ZNVIXORNO4MTXOVYDQZI4IJE4VXHZWC6JI35YKOR3UESSXJNTQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.067301676 (524h17m24.529394159s) [noise=true , address=45.91.116.9:4262 , piece_id=X53JZGRLZLX53UZLLCXXUQCFSQD2FO76X74OM45G26HNDU55UXXQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.067705735 (524h17m24.529798218s) [noise=true , address=145.79.227.4:4187 , piece_id=TQVPL6ZA6LHKAJVL6XWSYASZD3MCKKV3553CRFD72XWTVWY62STQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.069183369 (524h17m24.531275862s) [noise=true , address=145.79.227.5:4222 , piece_id=C2JSPN2QVSGSXG6ZW3ZMOFX5YFNN65XY7E62N5OEHOOMK3NSO4KQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.070534782 (524h17m24.532627265s) [noise=true , address=77.90.31.11:4555 , piece_id=U5RYZSD643KSUI4XLL26EAN6FQRPMG62GQMMCQRPEEG2XKSNBIZQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.072356982 (524h17m24.534449466s) [noise=true , address=145.79.227.11:4730 , piece_id=VO2DSM5LT43QAFPWOY3W3PVEJ44GZJWTCWG5QRQQFB363DFVFZDA, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.073961167 (524h17m24.53605364s) [noise=true , address=45.91.116.4:4940 , piece_id=3MCSUDYN73NIVJTGKPOVY6N3MALB3GBXI7I5NAX4HXPDHVMQ52BA, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.075716960 (524h17m24.537809433s) [noise=true , address=145.79.227.16:4910 , piece_id=BIHEJDH7RPI4UCPUNV453QHSEVCLA54PSNNQB2II37XZDRXFZNWQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.079266419 (524h17m24.541358902s) [noise=true , address=145.79.227.14:4838 , piece_id=6K6A4CNEHGB3X6DEBLF5KHLUT4GE6BQBW6IE3RHBML2NWPN255AA, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.080943032 (524h17m24.543035505s) [noise=true , address=45.91.116.12:4226 , piece_id=DRXRGO7OXEKVS5ACL6FC7RKLGBBD6KNI7X6E5DM5D5VV5PEKN7FQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.081340479 (524h17m24.543432962s) [noise=true , address=45.91.116.8:4045 , piece_id=EDXBNLKKRZMT6MKK2KOIZM7DDKA3IHB6UMCXX4YKZF7JY2T3S5CQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.083616223 (524h17m24.545708706s) [noise=true , address=145.79.227.14:4840 , piece_id=B4ZVPRMVH7Y3TCWCLPO7DWRVMJL7VSI73CIT4PEHCMTIEURQKB2Q, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.083633946 (524h17m24.545726429s) [noise=true , address=45.91.116.14:4189 , piece_id=UMQRQYPZ7SWUCO2N7JDCEVI5U5P36FDL3SUM6HIYJVWVOZTWNXQQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.085493247 (524h17m24.54758573s) [noise=true , address=145.79.227.17:4969 , piece_id=R5HIP33QW5HLICARFWDUOW7I75YTDHZTJCIML5X52VKHLPL3J4YQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:40.727263388 (524h17m26.189355871s) [upload=461218]: Next returned nil segment Oct 15 09:48:40.727310076 (524h17m26.189402549s) [upload=461218]: Waiting for error group managing segments... Oct 15 09:48:41.004514306 (524h17m26.466606789s) [noise=true , address=145.79.227.7:4294 , piece_id=H4ZNVIXORNO4MTXOVYDQZI4IJE4VXHZWC6JI35YKOR3UESSXJNTQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.006116557 (524h17m26.46820904s) [noise=true , address=145.79.227.5:4222 , piece_id=C2JSPN2QVSGSXG6ZW3ZMOFX5YFNN65XY7E62N5OEHOOMK3NSO4KQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.012963184 (524h17m26.475055667s) [noise=true , address=145.79.227.16:4910 , piece_id=BIHEJDH7RPI4UCPUNV453QHSEVCLA54PSNNQB2II37XZDRXFZNWQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.012994684 (524h17m26.475087167s) [noise=true , address=145.79.227.14:4840 , piece_id=B4ZVPRMVH7Y3TCWCLPO7DWRVMJL7VSI73CIT4PEHCMTIEURQKB2Q, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.016649956 (524h17m26.478742439s) [noise=true , address=145.79.227.11:4730 , piece_id=VO2DSM5LT43QAFPWOY3W3PVEJ44GZJWTCWG5QRQQFB363DFVFZDA, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.019248864 (524h17m26.481341347s) [noise=true , address=145.79.227.17:4969 , piece_id=R5HIP33QW5HLICARFWDUOW7I75YTDHZTJCIML5X52VKHLPL3J4YQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.075991585 (524h17m26.538084068s) [noise=true , address=145.79.227.4:4187 , piece_id=TQVPL6ZA6LHKAJVL6XWSYASZD3MCKKV3553CRFD72XWTVWY62STQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.118655146 (524h17m26.580747619s) [noise=true , address=145.79.227.14:4838 , piece_id=6K6A4CNEHGB3X6DEBLF5KHLUT4GE6BQBW6IE3RHBML2NWPN255AA, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.285934795 (524h17m26.748027278s) [noise=true , address=77.90.31.11:4555 , piece_id=U5RYZSD643KSUI4XLL26EAN6FQRPMG62GQMMCQRPEEG2XKSNBIZQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.744525465 (524h17m27.206617948s) [noise=true , address=45.91.116.9:4262 , piece_id=X53JZGRLZLX53UZLLCXXUQCFSQD2FO76X74OM45G26HNDU55UXXQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:42.218961989 (524h17m27.681054462s) [noise=true , address=45.91.116.11:4117 , piece_id=VH7ZJKSZ2NUXQJZE2IKNWP5PAL3TN2NNHD3ATZUQHGKKYMSSM3LQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:42.313638675 (524h17m27.775731158s) [noise=true , address=45.91.116.14:4189 , piece_id=UMQRQYPZ7SWUCO2N7JDCEVI5U5P36FDL3SUM6HIYJVWVOZTWNXQQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:42.434283559 (524h17m27.896376053s) [noise=true , address=45.91.116.8:4045 , piece_id=EDXBNLKKRZMT6MKK2KOIZM7DDKA3IHB6UMCXX4YKZF7JY2T3S5CQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:42.434321832 (524h17m27.896414305s) [seg_pos={0 0}, upload=461218]: Segment reached optimal threshold of 13 pieces. Oct 15 09:48:42.434605041 (524h17m27.896697534s) [noise=true , address=45.91.116.5:4299 , piece_id=4ARMXXE5LBRE2OZQEIITELKPTW4TISZWTD6XZGI4UJV7UCLUXAEQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX): protocol: expected piece hash; context canceled; EOF Oct 15 09:48:42.434666138 (524h17m27.896758621s) [noise=true , address=45.91.116.5:4299 , piece_id=4ARMXXE5LBRE2OZQEIITELKPTW4TISZWTD6XZGI4UJV7UCLUXAEQ, seg_pos={0 0}, upload=461218]: Long tail cancellation due to optimal threshold Oct 15 09:48:42.434692849 (524h17m27.896785432s) [noise=true , address=45.91.116.12:4226 , piece_id=DRXRGO7OXEKVS5ACL6FC7RKLGBBD6KNI7X6E5DM5D5VV5PEKN7FQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1): protocol: expected piece hash; context canceled; EOF Oct 15 09:48:42.434706184 (524h17m27.896798657s) [noise=true , address=45.91.116.12:4226 , piece_id=DRXRGO7OXEKVS5ACL6FC7RKLGBBD6KNI7X6E5DM5D5VV5PEKN7FQ, seg_pos={0 0}, upload=461218]: Long tail cancellation due to optimal threshold Oct 15 09:48:42.434718318 (524h17m27.896810911s) [noise=true , address=45.91.116.4:4940 , piece_id=3MCSUDYN73NIVJTGKPOVY6N3MALB3GBXI7I5NAX4HXPDHVMQ52BA, seg_pos={0 0}, upload=461218]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe): protocol: expected piece hash; context canceled; EOF Oct 15 09:48:42.434728447 (524h17m27.89682093s) [noise=true , address=45.91.116.4:4940 , piece_id=3MCSUDYN73NIVJTGKPOVY6N3MALB3GBXI7I5NAX4HXPDHVMQ52BA, seg_pos={0 0}, upload=461218]: Long tail cancellation due to optimal threshold Oct 15 09:48:42.434760458 (524h17m27.896852931s) [seg_pos={0 0}, upload=461218]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 15 09:48:42.434866900 (524h17m27.896959384s) [upload=461218]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 15 09:48:42.434873884 (524h17m27.896966367s) [upload=461218]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 15 09:48:42.760633614 (524h17m28.222726087s) [upload=461218]: Done uploading segments... Oct 15 09:48:42.760655435 (524h17m28.222747918s) [upload=461218]: Upload finished. err: <nil> Duration: 6.252936719s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [................X...X.......................................] duration: 6.236636111s FAIL (2025-10-15 10:17:48 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, Piece ID: VH7ZJKSZ2NUXQJZE2IKNWP5PAL3TN2NNHD3ATZUQHGKKYMSSM3LQ) context deadline exceeded; piecestore: (Node ID: 1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N, Piece ID: UMQRQYPZ7SWUCO2N7JDCEVI5U5P36FDL3SUM6HIYJVWVOZTWNXQQ) context deadline exceeded ======LOGS====== Oct 15 09:48:38.801058027 (524h17m24.26315051s) [upload=461218]: Starting upload Oct 15 09:48:38.801138119 (524h17m24.263230592s) [upload=461218]: Done starting upload Oct 15 09:48:38.801288545 (524h17m24.263381028s) [upload=461218]: Uploading segments... Oct 15 09:48:38.801442509 (524h17m24.263535002s) [upload=461218]: Got next segment. Inline: false Oct 15 09:48:38.801486663 (524h17m24.263579136s) [upload=461218]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 15 09:48:38.801493245 (524h17m24.263585728s) [upload=461218]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 15 09:48:39.066131918 (524h17m24.528224401s) [seg_pos={0 0}, upload=461218]: Begin upload segment... Oct 15 09:48:39.066205549 (524h17m24.528298022s) [seg_pos={0 0}, upload=461218]: Done begin upload segment. Oct 15 09:48:39.066243090 (524h17m24.528335573s) [noise=true , address=45.91.116.5:4299 , piece_id=4ARMXXE5LBRE2OZQEIITELKPTW4TISZWTD6XZGI4UJV7UCLUXAEQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.066254431 (524h17m24.528346915s) [noise=true , address=45.91.116.11:4117 , piece_id=VH7ZJKSZ2NUXQJZE2IKNWP5PAL3TN2NNHD3ATZUQHGKKYMSSM3LQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.066874873 (524h17m24.528967356s) [noise=true , address=145.79.227.7:4294 , piece_id=H4ZNVIXORNO4MTXOVYDQZI4IJE4VXHZWC6JI35YKOR3UESSXJNTQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.067301676 (524h17m24.529394159s) [noise=true , address=45.91.116.9:4262 , piece_id=X53JZGRLZLX53UZLLCXXUQCFSQD2FO76X74OM45G26HNDU55UXXQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.067705735 (524h17m24.529798218s) [noise=true , address=145.79.227.4:4187 , piece_id=TQVPL6ZA6LHKAJVL6XWSYASZD3MCKKV3553CRFD72XWTVWY62STQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.069183369 (524h17m24.531275862s) [noise=true , address=145.79.227.5:4222 , piece_id=C2JSPN2QVSGSXG6ZW3ZMOFX5YFNN65XY7E62N5OEHOOMK3NSO4KQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.070534782 (524h17m24.532627265s) [noise=true , address=77.90.31.11:4555 , piece_id=U5RYZSD643KSUI4XLL26EAN6FQRPMG62GQMMCQRPEEG2XKSNBIZQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.072356982 (524h17m24.534449466s) [noise=true , address=145.79.227.11:4730 , piece_id=VO2DSM5LT43QAFPWOY3W3PVEJ44GZJWTCWG5QRQQFB363DFVFZDA, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.073961167 (524h17m24.53605364s) [noise=true , address=45.91.116.4:4940 , piece_id=3MCSUDYN73NIVJTGKPOVY6N3MALB3GBXI7I5NAX4HXPDHVMQ52BA, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.075716960 (524h17m24.537809433s) [noise=true , address=145.79.227.16:4910 , piece_id=BIHEJDH7RPI4UCPUNV453QHSEVCLA54PSNNQB2II37XZDRXFZNWQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.079266419 (524h17m24.541358902s) [noise=true , address=145.79.227.14:4838 , piece_id=6K6A4CNEHGB3X6DEBLF5KHLUT4GE6BQBW6IE3RHBML2NWPN255AA, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.080943032 (524h17m24.543035505s) [noise=true , address=45.91.116.12:4226 , piece_id=DRXRGO7OXEKVS5ACL6FC7RKLGBBD6KNI7X6E5DM5D5VV5PEKN7FQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.081340479 (524h17m24.543432962s) [noise=true , address=45.91.116.8:4045 , piece_id=EDXBNLKKRZMT6MKK2KOIZM7DDKA3IHB6UMCXX4YKZF7JY2T3S5CQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.083616223 (524h17m24.545708706s) [noise=true , address=145.79.227.14:4840 , piece_id=B4ZVPRMVH7Y3TCWCLPO7DWRVMJL7VSI73CIT4PEHCMTIEURQKB2Q, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.083633946 (524h17m24.545726429s) [noise=true , address=45.91.116.14:4189 , piece_id=UMQRQYPZ7SWUCO2N7JDCEVI5U5P36FDL3SUM6HIYJVWVOZTWNXQQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.085493247 (524h17m24.54758573s) [noise=true , address=145.79.227.17:4969 , piece_id=R5HIP33QW5HLICARFWDUOW7I75YTDHZTJCIML5X52VKHLPL3J4YQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:40.727263388 (524h17m26.189355871s) [upload=461218]: Next returned nil segment Oct 15 09:48:40.727310076 (524h17m26.189402549s) [upload=461218]: Waiting for error group managing segments... Oct 15 09:48:41.004514306 (524h17m26.466606789s) [noise=true , address=145.79.227.7:4294 , piece_id=H4ZNVIXORNO4MTXOVYDQZI4IJE4VXHZWC6JI35YKOR3UESSXJNTQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.006116557 (524h17m26.46820904s) [noise=true , address=145.79.227.5:4222 , piece_id=C2JSPN2QVSGSXG6ZW3ZMOFX5YFNN65XY7E62N5OEHOOMK3NSO4KQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.012963184 (524h17m26.475055667s) [noise=true , address=145.79.227.16:4910 , piece_id=BIHEJDH7RPI4UCPUNV453QHSEVCLA54PSNNQB2II37XZDRXFZNWQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.012994684 (524h17m26.475087167s) [noise=true , address=145.79.227.14:4840 , piece_id=B4ZVPRMVH7Y3TCWCLPO7DWRVMJL7VSI73CIT4PEHCMTIEURQKB2Q, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.016649956 (524h17m26.478742439s) [noise=true , address=145.79.227.11:4730 , piece_id=VO2DSM5LT43QAFPWOY3W3PVEJ44GZJWTCWG5QRQQFB363DFVFZDA, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.019248864 (524h17m26.481341347s) [noise=true , address=145.79.227.17:4969 , piece_id=R5HIP33QW5HLICARFWDUOW7I75YTDHZTJCIML5X52VKHLPL3J4YQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.075991585 (524h17m26.538084068s) [noise=true , address=145.79.227.4:4187 , piece_id=TQVPL6ZA6LHKAJVL6XWSYASZD3MCKKV3553CRFD72XWTVWY62STQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.118655146 (524h17m26.580747619s) [noise=true , address=145.79.227.14:4838 , piece_id=6K6A4CNEHGB3X6DEBLF5KHLUT4GE6BQBW6IE3RHBML2NWPN255AA, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.285934795 (524h17m26.748027278s) [noise=true , address=77.90.31.11:4555 , piece_id=U5RYZSD643KSUI4XLL26EAN6FQRPMG62GQMMCQRPEEG2XKSNBIZQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.744525465 (524h17m27.206617948s) [noise=true , address=45.91.116.9:4262 , piece_id=X53JZGRLZLX53UZLLCXXUQCFSQD2FO76X74OM45G26HNDU55UXXQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:42.218961989 (524h17m27.681054462s) [noise=true , address=45.91.116.11:4117 , piece_id=VH7ZJKSZ2NUXQJZE2IKNWP5PAL3TN2NNHD3ATZUQHGKKYMSSM3LQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:42.313638675 (524h17m27.775731158s) [noise=true , address=45.91.116.14:4189 , piece_id=UMQRQYPZ7SWUCO2N7JDCEVI5U5P36FDL3SUM6HIYJVWVOZTWNXQQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:42.434283559 (524h17m27.896376053s) [noise=true , address=45.91.116.8:4045 , piece_id=EDXBNLKKRZMT6MKK2KOIZM7DDKA3IHB6UMCXX4YKZF7JY2T3S5CQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:42.434321832 (524h17m27.896414305s) [seg_pos={0 0}, upload=461218]: Segment reached optimal threshold of 13 pieces. Oct 15 09:48:42.434605041 (524h17m27.896697534s) [noise=true , address=45.91.116.5:4299 , piece_id=4ARMXXE5LBRE2OZQEIITELKPTW4TISZWTD6XZGI4UJV7UCLUXAEQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX): protocol: expected piece hash; context canceled; EOF Oct 15 09:48:42.434666138 (524h17m27.896758621s) [noise=true , address=45.91.116.5:4299 , piece_id=4ARMXXE5LBRE2OZQEIITELKPTW4TISZWTD6XZGI4UJV7UCLUXAEQ, seg_pos={0 0}, upload=461218]: Long tail cancellation due to optimal threshold Oct 15 09:48:42.434692849 (524h17m27.896785432s) [noise=true , address=45.91.116.12:4226 , piece_id=DRXRGO7OXEKVS5ACL6FC7RKLGBBD6KNI7X6E5DM5D5VV5PEKN7FQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1): protocol: expected piece hash; context canceled; EOF Oct 15 09:48:42.434706184 (524h17m27.896798657s) [noise=true , address=45.91.116.12:4226 , piece_id=DRXRGO7OXEKVS5ACL6FC7RKLGBBD6KNI7X6E5DM5D5VV5PEKN7FQ, seg_pos={0 0}, upload=461218]: Long tail cancellation due to optimal threshold Oct 15 09:48:42.434718318 (524h17m27.896810911s) [noise=true , address=45.91.116.4:4940 , piece_id=3MCSUDYN73NIVJTGKPOVY6N3MALB3GBXI7I5NAX4HXPDHVMQ52BA, seg_pos={0 0}, upload=461218]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe): protocol: expected piece hash; context canceled; EOF Oct 15 09:48:42.434728447 (524h17m27.89682093s) [noise=true , address=45.91.116.4:4940 , piece_id=3MCSUDYN73NIVJTGKPOVY6N3MALB3GBXI7I5NAX4HXPDHVMQ52BA, seg_pos={0 0}, upload=461218]: Long tail cancellation due to optimal threshold Oct 15 09:48:42.434760458 (524h17m27.896852931s) [seg_pos={0 0}, upload=461218]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 15 09:48:42.434866900 (524h17m27.896959384s) [upload=461218]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 15 09:48:42.434873884 (524h17m27.896966367s) [upload=461218]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 15 09:48:42.760633614 (524h17m28.222726087s) [upload=461218]: Done uploading segments... Oct 15 09:48:42.760655435 (524h17m28.222747918s) [upload=461218]: Upload finished. err: <nil> Duration: 6.2366732s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.................X...X......................................] duration: 6.369083012s FAIL (2025-10-15 10:16:49 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, Piece ID: VH7ZJKSZ2NUXQJZE2IKNWP5PAL3TN2NNHD3ATZUQHGKKYMSSM3LQ) context deadline exceeded; piecestore: (Node ID: 1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N, Piece ID: UMQRQYPZ7SWUCO2N7JDCEVI5U5P36FDL3SUM6HIYJVWVOZTWNXQQ) context deadline exceeded ======LOGS====== Oct 15 09:48:38.801058027 (524h17m24.26315051s) [upload=461218]: Starting upload Oct 15 09:48:38.801138119 (524h17m24.263230592s) [upload=461218]: Done starting upload Oct 15 09:48:38.801288545 (524h17m24.263381028s) [upload=461218]: Uploading segments... Oct 15 09:48:38.801442509 (524h17m24.263535002s) [upload=461218]: Got next segment. Inline: false Oct 15 09:48:38.801486663 (524h17m24.263579136s) [upload=461218]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 15 09:48:38.801493245 (524h17m24.263585728s) [upload=461218]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 15 09:48:39.066131918 (524h17m24.528224401s) [seg_pos={0 0}, upload=461218]: Begin upload segment... Oct 15 09:48:39.066205549 (524h17m24.528298022s) [seg_pos={0 0}, upload=461218]: Done begin upload segment. Oct 15 09:48:39.066243090 (524h17m24.528335573s) [noise=true , address=45.91.116.5:4299 , piece_id=4ARMXXE5LBRE2OZQEIITELKPTW4TISZWTD6XZGI4UJV7UCLUXAEQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.066254431 (524h17m24.528346915s) [noise=true , address=45.91.116.11:4117 , piece_id=VH7ZJKSZ2NUXQJZE2IKNWP5PAL3TN2NNHD3ATZUQHGKKYMSSM3LQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.066874873 (524h17m24.528967356s) [noise=true , address=145.79.227.7:4294 , piece_id=H4ZNVIXORNO4MTXOVYDQZI4IJE4VXHZWC6JI35YKOR3UESSXJNTQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.067301676 (524h17m24.529394159s) [noise=true , address=45.91.116.9:4262 , piece_id=X53JZGRLZLX53UZLLCXXUQCFSQD2FO76X74OM45G26HNDU55UXXQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.067705735 (524h17m24.529798218s) [noise=true , address=145.79.227.4:4187 , piece_id=TQVPL6ZA6LHKAJVL6XWSYASZD3MCKKV3553CRFD72XWTVWY62STQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.069183369 (524h17m24.531275862s) [noise=true , address=145.79.227.5:4222 , piece_id=C2JSPN2QVSGSXG6ZW3ZMOFX5YFNN65XY7E62N5OEHOOMK3NSO4KQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.070534782 (524h17m24.532627265s) [noise=true , address=77.90.31.11:4555 , piece_id=U5RYZSD643KSUI4XLL26EAN6FQRPMG62GQMMCQRPEEG2XKSNBIZQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.072356982 (524h17m24.534449466s) [noise=true , address=145.79.227.11:4730 , piece_id=VO2DSM5LT43QAFPWOY3W3PVEJ44GZJWTCWG5QRQQFB363DFVFZDA, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.073961167 (524h17m24.53605364s) [noise=true , address=45.91.116.4:4940 , piece_id=3MCSUDYN73NIVJTGKPOVY6N3MALB3GBXI7I5NAX4HXPDHVMQ52BA, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.075716960 (524h17m24.537809433s) [noise=true , address=145.79.227.16:4910 , piece_id=BIHEJDH7RPI4UCPUNV453QHSEVCLA54PSNNQB2II37XZDRXFZNWQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.079266419 (524h17m24.541358902s) [noise=true , address=145.79.227.14:4838 , piece_id=6K6A4CNEHGB3X6DEBLF5KHLUT4GE6BQBW6IE3RHBML2NWPN255AA, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.080943032 (524h17m24.543035505s) [noise=true , address=45.91.116.12:4226 , piece_id=DRXRGO7OXEKVS5ACL6FC7RKLGBBD6KNI7X6E5DM5D5VV5PEKN7FQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.081340479 (524h17m24.543432962s) [noise=true , address=45.91.116.8:4045 , piece_id=EDXBNLKKRZMT6MKK2KOIZM7DDKA3IHB6UMCXX4YKZF7JY2T3S5CQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.083616223 (524h17m24.545708706s) [noise=true , address=145.79.227.14:4840 , piece_id=B4ZVPRMVH7Y3TCWCLPO7DWRVMJL7VSI73CIT4PEHCMTIEURQKB2Q, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.083633946 (524h17m24.545726429s) [noise=true , address=45.91.116.14:4189 , piece_id=UMQRQYPZ7SWUCO2N7JDCEVI5U5P36FDL3SUM6HIYJVWVOZTWNXQQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.085493247 (524h17m24.54758573s) [noise=true , address=145.79.227.17:4969 , piece_id=R5HIP33QW5HLICARFWDUOW7I75YTDHZTJCIML5X52VKHLPL3J4YQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:40.727263388 (524h17m26.189355871s) [upload=461218]: Next returned nil segment Oct 15 09:48:40.727310076 (524h17m26.189402549s) [upload=461218]: Waiting for error group managing segments... Oct 15 09:48:41.004514306 (524h17m26.466606789s) [noise=true , address=145.79.227.7:4294 , piece_id=H4ZNVIXORNO4MTXOVYDQZI4IJE4VXHZWC6JI35YKOR3UESSXJNTQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.006116557 (524h17m26.46820904s) [noise=true , address=145.79.227.5:4222 , piece_id=C2JSPN2QVSGSXG6ZW3ZMOFX5YFNN65XY7E62N5OEHOOMK3NSO4KQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.012963184 (524h17m26.475055667s) [noise=true , address=145.79.227.16:4910 , piece_id=BIHEJDH7RPI4UCPUNV453QHSEVCLA54PSNNQB2II37XZDRXFZNWQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.012994684 (524h17m26.475087167s) [noise=true , address=145.79.227.14:4840 , piece_id=B4ZVPRMVH7Y3TCWCLPO7DWRVMJL7VSI73CIT4PEHCMTIEURQKB2Q, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.016649956 (524h17m26.478742439s) [noise=true , address=145.79.227.11:4730 , piece_id=VO2DSM5LT43QAFPWOY3W3PVEJ44GZJWTCWG5QRQQFB363DFVFZDA, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.019248864 (524h17m26.481341347s) [noise=true , address=145.79.227.17:4969 , piece_id=R5HIP33QW5HLICARFWDUOW7I75YTDHZTJCIML5X52VKHLPL3J4YQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.075991585 (524h17m26.538084068s) [noise=true , address=145.79.227.4:4187 , piece_id=TQVPL6ZA6LHKAJVL6XWSYASZD3MCKKV3553CRFD72XWTVWY62STQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.118655146 (524h17m26.580747619s) [noise=true , address=145.79.227.14:4838 , piece_id=6K6A4CNEHGB3X6DEBLF5KHLUT4GE6BQBW6IE3RHBML2NWPN255AA, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.285934795 (524h17m26.748027278s) [noise=true , address=77.90.31.11:4555 , piece_id=U5RYZSD643KSUI4XLL26EAN6FQRPMG62GQMMCQRPEEG2XKSNBIZQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.744525465 (524h17m27.206617948s) [noise=true , address=45.91.116.9:4262 , piece_id=X53JZGRLZLX53UZLLCXXUQCFSQD2FO76X74OM45G26HNDU55UXXQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:42.218961989 (524h17m27.681054462s) [noise=true , address=45.91.116.11:4117 , piece_id=VH7ZJKSZ2NUXQJZE2IKNWP5PAL3TN2NNHD3ATZUQHGKKYMSSM3LQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:42.313638675 (524h17m27.775731158s) [noise=true , address=45.91.116.14:4189 , piece_id=UMQRQYPZ7SWUCO2N7JDCEVI5U5P36FDL3SUM6HIYJVWVOZTWNXQQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:42.434283559 (524h17m27.896376053s) [noise=true , address=45.91.116.8:4045 , piece_id=EDXBNLKKRZMT6MKK2KOIZM7DDKA3IHB6UMCXX4YKZF7JY2T3S5CQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:42.434321832 (524h17m27.896414305s) [seg_pos={0 0}, upload=461218]: Segment reached optimal threshold of 13 pieces. Oct 15 09:48:42.434605041 (524h17m27.896697534s) [noise=true , address=45.91.116.5:4299 , piece_id=4ARMXXE5LBRE2OZQEIITELKPTW4TISZWTD6XZGI4UJV7UCLUXAEQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX): protocol: expected piece hash; context canceled; EOF Oct 15 09:48:42.434666138 (524h17m27.896758621s) [noise=true , address=45.91.116.5:4299 , piece_id=4ARMXXE5LBRE2OZQEIITELKPTW4TISZWTD6XZGI4UJV7UCLUXAEQ, seg_pos={0 0}, upload=461218]: Long tail cancellation due to optimal threshold Oct 15 09:48:42.434692849 (524h17m27.896785432s) [noise=true , address=45.91.116.12:4226 , piece_id=DRXRGO7OXEKVS5ACL6FC7RKLGBBD6KNI7X6E5DM5D5VV5PEKN7FQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1): protocol: expected piece hash; context canceled; EOF Oct 15 09:48:42.434706184 (524h17m27.896798657s) [noise=true , address=45.91.116.12:4226 , piece_id=DRXRGO7OXEKVS5ACL6FC7RKLGBBD6KNI7X6E5DM5D5VV5PEKN7FQ, seg_pos={0 0}, upload=461218]: Long tail cancellation due to optimal threshold Oct 15 09:48:42.434718318 (524h17m27.896810911s) [noise=true , address=45.91.116.4:4940 , piece_id=3MCSUDYN73NIVJTGKPOVY6N3MALB3GBXI7I5NAX4HXPDHVMQ52BA, seg_pos={0 0}, upload=461218]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe): protocol: expected piece hash; context canceled; EOF Oct 15 09:48:42.434728447 (524h17m27.89682093s) [noise=true , address=45.91.116.4:4940 , piece_id=3MCSUDYN73NIVJTGKPOVY6N3MALB3GBXI7I5NAX4HXPDHVMQ52BA, seg_pos={0 0}, upload=461218]: Long tail cancellation due to optimal threshold Oct 15 09:48:42.434760458 (524h17m27.896852931s) [seg_pos={0 0}, upload=461218]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 15 09:48:42.434866900 (524h17m27.896959384s) [upload=461218]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 15 09:48:42.434873884 (524h17m27.896966367s) [upload=461218]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 15 09:48:42.760633614 (524h17m28.222726087s) [upload=461218]: Done uploading segments... Oct 15 09:48:42.760655435 (524h17m28.222747918s) [upload=461218]: Upload finished. err: <nil> Duration: 6.369112288s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..................X...X.....................................] duration: 6.137388619s FAIL (2025-10-15 10:15:40 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, Piece ID: VH7ZJKSZ2NUXQJZE2IKNWP5PAL3TN2NNHD3ATZUQHGKKYMSSM3LQ) context deadline exceeded; piecestore: (Node ID: 1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N, Piece ID: UMQRQYPZ7SWUCO2N7JDCEVI5U5P36FDL3SUM6HIYJVWVOZTWNXQQ) context deadline exceeded ======LOGS====== Oct 15 09:48:38.801058027 (524h17m24.26315051s) [upload=461218]: Starting upload Oct 15 09:48:38.801138119 (524h17m24.263230592s) [upload=461218]: Done starting upload Oct 15 09:48:38.801288545 (524h17m24.263381028s) [upload=461218]: Uploading segments... Oct 15 09:48:38.801442509 (524h17m24.263535002s) [upload=461218]: Got next segment. Inline: false Oct 15 09:48:38.801486663 (524h17m24.263579136s) [upload=461218]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 15 09:48:38.801493245 (524h17m24.263585728s) [upload=461218]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 15 09:48:39.066131918 (524h17m24.528224401s) [seg_pos={0 0}, upload=461218]: Begin upload segment... Oct 15 09:48:39.066205549 (524h17m24.528298022s) [seg_pos={0 0}, upload=461218]: Done begin upload segment. Oct 15 09:48:39.066243090 (524h17m24.528335573s) [noise=true , address=45.91.116.5:4299 , piece_id=4ARMXXE5LBRE2OZQEIITELKPTW4TISZWTD6XZGI4UJV7UCLUXAEQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.066254431 (524h17m24.528346915s) [noise=true , address=45.91.116.11:4117 , piece_id=VH7ZJKSZ2NUXQJZE2IKNWP5PAL3TN2NNHD3ATZUQHGKKYMSSM3LQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.066874873 (524h17m24.528967356s) [noise=true , address=145.79.227.7:4294 , piece_id=H4ZNVIXORNO4MTXOVYDQZI4IJE4VXHZWC6JI35YKOR3UESSXJNTQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.067301676 (524h17m24.529394159s) [noise=true , address=45.91.116.9:4262 , piece_id=X53JZGRLZLX53UZLLCXXUQCFSQD2FO76X74OM45G26HNDU55UXXQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.067705735 (524h17m24.529798218s) [noise=true , address=145.79.227.4:4187 , piece_id=TQVPL6ZA6LHKAJVL6XWSYASZD3MCKKV3553CRFD72XWTVWY62STQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.069183369 (524h17m24.531275862s) [noise=true , address=145.79.227.5:4222 , piece_id=C2JSPN2QVSGSXG6ZW3ZMOFX5YFNN65XY7E62N5OEHOOMK3NSO4KQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.070534782 (524h17m24.532627265s) [noise=true , address=77.90.31.11:4555 , piece_id=U5RYZSD643KSUI4XLL26EAN6FQRPMG62GQMMCQRPEEG2XKSNBIZQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.072356982 (524h17m24.534449466s) [noise=true , address=145.79.227.11:4730 , piece_id=VO2DSM5LT43QAFPWOY3W3PVEJ44GZJWTCWG5QRQQFB363DFVFZDA, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.073961167 (524h17m24.53605364s) [noise=true , address=45.91.116.4:4940 , piece_id=3MCSUDYN73NIVJTGKPOVY6N3MALB3GBXI7I5NAX4HXPDHVMQ52BA, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.075716960 (524h17m24.537809433s) [noise=true , address=145.79.227.16:4910 , piece_id=BIHEJDH7RPI4UCPUNV453QHSEVCLA54PSNNQB2II37XZDRXFZNWQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.079266419 (524h17m24.541358902s) [noise=true , address=145.79.227.14:4838 , piece_id=6K6A4CNEHGB3X6DEBLF5KHLUT4GE6BQBW6IE3RHBML2NWPN255AA, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.080943032 (524h17m24.543035505s) [noise=true , address=45.91.116.12:4226 , piece_id=DRXRGO7OXEKVS5ACL6FC7RKLGBBD6KNI7X6E5DM5D5VV5PEKN7FQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.081340479 (524h17m24.543432962s) [noise=true , address=45.91.116.8:4045 , piece_id=EDXBNLKKRZMT6MKK2KOIZM7DDKA3IHB6UMCXX4YKZF7JY2T3S5CQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.083616223 (524h17m24.545708706s) [noise=true , address=145.79.227.14:4840 , piece_id=B4ZVPRMVH7Y3TCWCLPO7DWRVMJL7VSI73CIT4PEHCMTIEURQKB2Q, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.083633946 (524h17m24.545726429s) [noise=true , address=45.91.116.14:4189 , piece_id=UMQRQYPZ7SWUCO2N7JDCEVI5U5P36FDL3SUM6HIYJVWVOZTWNXQQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.085493247 (524h17m24.54758573s) [noise=true , address=145.79.227.17:4969 , piece_id=R5HIP33QW5HLICARFWDUOW7I75YTDHZTJCIML5X52VKHLPL3J4YQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:40.727263388 (524h17m26.189355871s) [upload=461218]: Next returned nil segment Oct 15 09:48:40.727310076 (524h17m26.189402549s) [upload=461218]: Waiting for error group managing segments... Oct 15 09:48:41.004514306 (524h17m26.466606789s) [noise=true , address=145.79.227.7:4294 , piece_id=H4ZNVIXORNO4MTXOVYDQZI4IJE4VXHZWC6JI35YKOR3UESSXJNTQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.006116557 (524h17m26.46820904s) [noise=true , address=145.79.227.5:4222 , piece_id=C2JSPN2QVSGSXG6ZW3ZMOFX5YFNN65XY7E62N5OEHOOMK3NSO4KQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.012963184 (524h17m26.475055667s) [noise=true , address=145.79.227.16:4910 , piece_id=BIHEJDH7RPI4UCPUNV453QHSEVCLA54PSNNQB2II37XZDRXFZNWQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.012994684 (524h17m26.475087167s) [noise=true , address=145.79.227.14:4840 , piece_id=B4ZVPRMVH7Y3TCWCLPO7DWRVMJL7VSI73CIT4PEHCMTIEURQKB2Q, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.016649956 (524h17m26.478742439s) [noise=true , address=145.79.227.11:4730 , piece_id=VO2DSM5LT43QAFPWOY3W3PVEJ44GZJWTCWG5QRQQFB363DFVFZDA, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.019248864 (524h17m26.481341347s) [noise=true , address=145.79.227.17:4969 , piece_id=R5HIP33QW5HLICARFWDUOW7I75YTDHZTJCIML5X52VKHLPL3J4YQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.075991585 (524h17m26.538084068s) [noise=true , address=145.79.227.4:4187 , piece_id=TQVPL6ZA6LHKAJVL6XWSYASZD3MCKKV3553CRFD72XWTVWY62STQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.118655146 (524h17m26.580747619s) [noise=true , address=145.79.227.14:4838 , piece_id=6K6A4CNEHGB3X6DEBLF5KHLUT4GE6BQBW6IE3RHBML2NWPN255AA, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.285934795 (524h17m26.748027278s) [noise=true , address=77.90.31.11:4555 , piece_id=U5RYZSD643KSUI4XLL26EAN6FQRPMG62GQMMCQRPEEG2XKSNBIZQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.744525465 (524h17m27.206617948s) [noise=true , address=45.91.116.9:4262 , piece_id=X53JZGRLZLX53UZLLCXXUQCFSQD2FO76X74OM45G26HNDU55UXXQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:42.218961989 (524h17m27.681054462s) [noise=true , address=45.91.116.11:4117 , piece_id=VH7ZJKSZ2NUXQJZE2IKNWP5PAL3TN2NNHD3ATZUQHGKKYMSSM3LQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:42.313638675 (524h17m27.775731158s) [noise=true , address=45.91.116.14:4189 , piece_id=UMQRQYPZ7SWUCO2N7JDCEVI5U5P36FDL3SUM6HIYJVWVOZTWNXQQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:42.434283559 (524h17m27.896376053s) [noise=true , address=45.91.116.8:4045 , piece_id=EDXBNLKKRZMT6MKK2KOIZM7DDKA3IHB6UMCXX4YKZF7JY2T3S5CQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:42.434321832 (524h17m27.896414305s) [seg_pos={0 0}, upload=461218]: Segment reached optimal threshold of 13 pieces. Oct 15 09:48:42.434605041 (524h17m27.896697534s) [noise=true , address=45.91.116.5:4299 , piece_id=4ARMXXE5LBRE2OZQEIITELKPTW4TISZWTD6XZGI4UJV7UCLUXAEQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX): protocol: expected piece hash; context canceled; EOF Oct 15 09:48:42.434666138 (524h17m27.896758621s) [noise=true , address=45.91.116.5:4299 , piece_id=4ARMXXE5LBRE2OZQEIITELKPTW4TISZWTD6XZGI4UJV7UCLUXAEQ, seg_pos={0 0}, upload=461218]: Long tail cancellation due to optimal threshold Oct 15 09:48:42.434692849 (524h17m27.896785432s) [noise=true , address=45.91.116.12:4226 , piece_id=DRXRGO7OXEKVS5ACL6FC7RKLGBBD6KNI7X6E5DM5D5VV5PEKN7FQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1): protocol: expected piece hash; context canceled; EOF Oct 15 09:48:42.434706184 (524h17m27.896798657s) [noise=true , address=45.91.116.12:4226 , piece_id=DRXRGO7OXEKVS5ACL6FC7RKLGBBD6KNI7X6E5DM5D5VV5PEKN7FQ, seg_pos={0 0}, upload=461218]: Long tail cancellation due to optimal threshold Oct 15 09:48:42.434718318 (524h17m27.896810911s) [noise=true , address=45.91.116.4:4940 , piece_id=3MCSUDYN73NIVJTGKPOVY6N3MALB3GBXI7I5NAX4HXPDHVMQ52BA, seg_pos={0 0}, upload=461218]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe): protocol: expected piece hash; context canceled; EOF Oct 15 09:48:42.434728447 (524h17m27.89682093s) [noise=true , address=45.91.116.4:4940 , piece_id=3MCSUDYN73NIVJTGKPOVY6N3MALB3GBXI7I5NAX4HXPDHVMQ52BA, seg_pos={0 0}, upload=461218]: Long tail cancellation due to optimal threshold Oct 15 09:48:42.434760458 (524h17m27.896852931s) [seg_pos={0 0}, upload=461218]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 15 09:48:42.434866900 (524h17m27.896959384s) [upload=461218]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 15 09:48:42.434873884 (524h17m27.896966367s) [upload=461218]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 15 09:48:42.760633614 (524h17m28.222726087s) [upload=461218]: Done uploading segments... Oct 15 09:48:42.760655435 (524h17m28.222747918s) [upload=461218]: Upload finished. err: <nil> Duration: 6.137414238s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...................X...X....................................] duration: 5.954898368s FAIL (2025-10-15 10:14:25 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, Piece ID: VH7ZJKSZ2NUXQJZE2IKNWP5PAL3TN2NNHD3ATZUQHGKKYMSSM3LQ) context deadline exceeded; piecestore: (Node ID: 1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N, Piece ID: UMQRQYPZ7SWUCO2N7JDCEVI5U5P36FDL3SUM6HIYJVWVOZTWNXQQ) context deadline exceeded ======LOGS====== Oct 15 09:48:38.801058027 (524h17m24.26315051s) [upload=461218]: Starting upload Oct 15 09:48:38.801138119 (524h17m24.263230592s) [upload=461218]: Done starting upload Oct 15 09:48:38.801288545 (524h17m24.263381028s) [upload=461218]: Uploading segments... Oct 15 09:48:38.801442509 (524h17m24.263535002s) [upload=461218]: Got next segment. Inline: false Oct 15 09:48:38.801486663 (524h17m24.263579136s) [upload=461218]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 15 09:48:38.801493245 (524h17m24.263585728s) [upload=461218]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 15 09:48:39.066131918 (524h17m24.528224401s) [seg_pos={0 0}, upload=461218]: Begin upload segment... Oct 15 09:48:39.066205549 (524h17m24.528298022s) [seg_pos={0 0}, upload=461218]: Done begin upload segment. Oct 15 09:48:39.066243090 (524h17m24.528335573s) [noise=true , address=45.91.116.5:4299 , piece_id=4ARMXXE5LBRE2OZQEIITELKPTW4TISZWTD6XZGI4UJV7UCLUXAEQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.066254431 (524h17m24.528346915s) [noise=true , address=45.91.116.11:4117 , piece_id=VH7ZJKSZ2NUXQJZE2IKNWP5PAL3TN2NNHD3ATZUQHGKKYMSSM3LQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.066874873 (524h17m24.528967356s) [noise=true , address=145.79.227.7:4294 , piece_id=H4ZNVIXORNO4MTXOVYDQZI4IJE4VXHZWC6JI35YKOR3UESSXJNTQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.067301676 (524h17m24.529394159s) [noise=true , address=45.91.116.9:4262 , piece_id=X53JZGRLZLX53UZLLCXXUQCFSQD2FO76X74OM45G26HNDU55UXXQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.067705735 (524h17m24.529798218s) [noise=true , address=145.79.227.4:4187 , piece_id=TQVPL6ZA6LHKAJVL6XWSYASZD3MCKKV3553CRFD72XWTVWY62STQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.069183369 (524h17m24.531275862s) [noise=true , address=145.79.227.5:4222 , piece_id=C2JSPN2QVSGSXG6ZW3ZMOFX5YFNN65XY7E62N5OEHOOMK3NSO4KQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.070534782 (524h17m24.532627265s) [noise=true , address=77.90.31.11:4555 , piece_id=U5RYZSD643KSUI4XLL26EAN6FQRPMG62GQMMCQRPEEG2XKSNBIZQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.072356982 (524h17m24.534449466s) [noise=true , address=145.79.227.11:4730 , piece_id=VO2DSM5LT43QAFPWOY3W3PVEJ44GZJWTCWG5QRQQFB363DFVFZDA, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.073961167 (524h17m24.53605364s) [noise=true , address=45.91.116.4:4940 , piece_id=3MCSUDYN73NIVJTGKPOVY6N3MALB3GBXI7I5NAX4HXPDHVMQ52BA, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.075716960 (524h17m24.537809433s) [noise=true , address=145.79.227.16:4910 , piece_id=BIHEJDH7RPI4UCPUNV453QHSEVCLA54PSNNQB2II37XZDRXFZNWQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.079266419 (524h17m24.541358902s) [noise=true , address=145.79.227.14:4838 , piece_id=6K6A4CNEHGB3X6DEBLF5KHLUT4GE6BQBW6IE3RHBML2NWPN255AA, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.080943032 (524h17m24.543035505s) [noise=true , address=45.91.116.12:4226 , piece_id=DRXRGO7OXEKVS5ACL6FC7RKLGBBD6KNI7X6E5DM5D5VV5PEKN7FQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.081340479 (524h17m24.543432962s) [noise=true , address=45.91.116.8:4045 , piece_id=EDXBNLKKRZMT6MKK2KOIZM7DDKA3IHB6UMCXX4YKZF7JY2T3S5CQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.083616223 (524h17m24.545708706s) [noise=true , address=145.79.227.14:4840 , piece_id=B4ZVPRMVH7Y3TCWCLPO7DWRVMJL7VSI73CIT4PEHCMTIEURQKB2Q, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.083633946 (524h17m24.545726429s) [noise=true , address=45.91.116.14:4189 , piece_id=UMQRQYPZ7SWUCO2N7JDCEVI5U5P36FDL3SUM6HIYJVWVOZTWNXQQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.085493247 (524h17m24.54758573s) [noise=true , address=145.79.227.17:4969 , piece_id=R5HIP33QW5HLICARFWDUOW7I75YTDHZTJCIML5X52VKHLPL3J4YQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:40.727263388 (524h17m26.189355871s) [upload=461218]: Next returned nil segment Oct 15 09:48:40.727310076 (524h17m26.189402549s) [upload=461218]: Waiting for error group managing segments... Oct 15 09:48:41.004514306 (524h17m26.466606789s) [noise=true , address=145.79.227.7:4294 , piece_id=H4ZNVIXORNO4MTXOVYDQZI4IJE4VXHZWC6JI35YKOR3UESSXJNTQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.006116557 (524h17m26.46820904s) [noise=true , address=145.79.227.5:4222 , piece_id=C2JSPN2QVSGSXG6ZW3ZMOFX5YFNN65XY7E62N5OEHOOMK3NSO4KQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.012963184 (524h17m26.475055667s) [noise=true , address=145.79.227.16:4910 , piece_id=BIHEJDH7RPI4UCPUNV453QHSEVCLA54PSNNQB2II37XZDRXFZNWQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.012994684 (524h17m26.475087167s) [noise=true , address=145.79.227.14:4840 , piece_id=B4ZVPRMVH7Y3TCWCLPO7DWRVMJL7VSI73CIT4PEHCMTIEURQKB2Q, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.016649956 (524h17m26.478742439s) [noise=true , address=145.79.227.11:4730 , piece_id=VO2DSM5LT43QAFPWOY3W3PVEJ44GZJWTCWG5QRQQFB363DFVFZDA, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.019248864 (524h17m26.481341347s) [noise=true , address=145.79.227.17:4969 , piece_id=R5HIP33QW5HLICARFWDUOW7I75YTDHZTJCIML5X52VKHLPL3J4YQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.075991585 (524h17m26.538084068s) [noise=true , address=145.79.227.4:4187 , piece_id=TQVPL6ZA6LHKAJVL6XWSYASZD3MCKKV3553CRFD72XWTVWY62STQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.118655146 (524h17m26.580747619s) [noise=true , address=145.79.227.14:4838 , piece_id=6K6A4CNEHGB3X6DEBLF5KHLUT4GE6BQBW6IE3RHBML2NWPN255AA, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.285934795 (524h17m26.748027278s) [noise=true , address=77.90.31.11:4555 , piece_id=U5RYZSD643KSUI4XLL26EAN6FQRPMG62GQMMCQRPEEG2XKSNBIZQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.744525465 (524h17m27.206617948s) [noise=true , address=45.91.116.9:4262 , piece_id=X53JZGRLZLX53UZLLCXXUQCFSQD2FO76X74OM45G26HNDU55UXXQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:42.218961989 (524h17m27.681054462s) [noise=true , address=45.91.116.11:4117 , piece_id=VH7ZJKSZ2NUXQJZE2IKNWP5PAL3TN2NNHD3ATZUQHGKKYMSSM3LQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:42.313638675 (524h17m27.775731158s) [noise=true , address=45.91.116.14:4189 , piece_id=UMQRQYPZ7SWUCO2N7JDCEVI5U5P36FDL3SUM6HIYJVWVOZTWNXQQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:42.434283559 (524h17m27.896376053s) [noise=true , address=45.91.116.8:4045 , piece_id=EDXBNLKKRZMT6MKK2KOIZM7DDKA3IHB6UMCXX4YKZF7JY2T3S5CQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:42.434321832 (524h17m27.896414305s) [seg_pos={0 0}, upload=461218]: Segment reached optimal threshold of 13 pieces. Oct 15 09:48:42.434605041 (524h17m27.896697534s) [noise=true , address=45.91.116.5:4299 , piece_id=4ARMXXE5LBRE2OZQEIITELKPTW4TISZWTD6XZGI4UJV7UCLUXAEQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX): protocol: expected piece hash; context canceled; EOF Oct 15 09:48:42.434666138 (524h17m27.896758621s) [noise=true , address=45.91.116.5:4299 , piece_id=4ARMXXE5LBRE2OZQEIITELKPTW4TISZWTD6XZGI4UJV7UCLUXAEQ, seg_pos={0 0}, upload=461218]: Long tail cancellation due to optimal threshold Oct 15 09:48:42.434692849 (524h17m27.896785432s) [noise=true , address=45.91.116.12:4226 , piece_id=DRXRGO7OXEKVS5ACL6FC7RKLGBBD6KNI7X6E5DM5D5VV5PEKN7FQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1): protocol: expected piece hash; context canceled; EOF Oct 15 09:48:42.434706184 (524h17m27.896798657s) [noise=true , address=45.91.116.12:4226 , piece_id=DRXRGO7OXEKVS5ACL6FC7RKLGBBD6KNI7X6E5DM5D5VV5PEKN7FQ, seg_pos={0 0}, upload=461218]: Long tail cancellation due to optimal threshold Oct 15 09:48:42.434718318 (524h17m27.896810911s) [noise=true , address=45.91.116.4:4940 , piece_id=3MCSUDYN73NIVJTGKPOVY6N3MALB3GBXI7I5NAX4HXPDHVMQ52BA, seg_pos={0 0}, upload=461218]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe): protocol: expected piece hash; context canceled; EOF Oct 15 09:48:42.434728447 (524h17m27.89682093s) [noise=true , address=45.91.116.4:4940 , piece_id=3MCSUDYN73NIVJTGKPOVY6N3MALB3GBXI7I5NAX4HXPDHVMQ52BA, seg_pos={0 0}, upload=461218]: Long tail cancellation due to optimal threshold Oct 15 09:48:42.434760458 (524h17m27.896852931s) [seg_pos={0 0}, upload=461218]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 15 09:48:42.434866900 (524h17m27.896959384s) [upload=461218]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 15 09:48:42.434873884 (524h17m27.896966367s) [upload=461218]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 15 09:48:42.760633614 (524h17m28.222726087s) [upload=461218]: Done uploading segments... Oct 15 09:48:42.760655435 (524h17m28.222747918s) [upload=461218]: Upload finished. err: <nil> Duration: 5.954936469s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [....................X...X...................................] duration: 6.162930279s FAIL (2025-10-15 10:13:18 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, Piece ID: VH7ZJKSZ2NUXQJZE2IKNWP5PAL3TN2NNHD3ATZUQHGKKYMSSM3LQ) context deadline exceeded; piecestore: (Node ID: 1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N, Piece ID: UMQRQYPZ7SWUCO2N7JDCEVI5U5P36FDL3SUM6HIYJVWVOZTWNXQQ) context deadline exceeded ======LOGS====== Oct 15 09:48:38.801058027 (524h17m24.26315051s) [upload=461218]: Starting upload Oct 15 09:48:38.801138119 (524h17m24.263230592s) [upload=461218]: Done starting upload Oct 15 09:48:38.801288545 (524h17m24.263381028s) [upload=461218]: Uploading segments... Oct 15 09:48:38.801442509 (524h17m24.263535002s) [upload=461218]: Got next segment. Inline: false Oct 15 09:48:38.801486663 (524h17m24.263579136s) [upload=461218]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 15 09:48:38.801493245 (524h17m24.263585728s) [upload=461218]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 15 09:48:39.066131918 (524h17m24.528224401s) [seg_pos={0 0}, upload=461218]: Begin upload segment... Oct 15 09:48:39.066205549 (524h17m24.528298022s) [seg_pos={0 0}, upload=461218]: Done begin upload segment. Oct 15 09:48:39.066243090 (524h17m24.528335573s) [noise=true , address=45.91.116.5:4299 , piece_id=4ARMXXE5LBRE2OZQEIITELKPTW4TISZWTD6XZGI4UJV7UCLUXAEQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.066254431 (524h17m24.528346915s) [noise=true , address=45.91.116.11:4117 , piece_id=VH7ZJKSZ2NUXQJZE2IKNWP5PAL3TN2NNHD3ATZUQHGKKYMSSM3LQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.066874873 (524h17m24.528967356s) [noise=true , address=145.79.227.7:4294 , piece_id=H4ZNVIXORNO4MTXOVYDQZI4IJE4VXHZWC6JI35YKOR3UESSXJNTQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.067301676 (524h17m24.529394159s) [noise=true , address=45.91.116.9:4262 , piece_id=X53JZGRLZLX53UZLLCXXUQCFSQD2FO76X74OM45G26HNDU55UXXQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.067705735 (524h17m24.529798218s) [noise=true , address=145.79.227.4:4187 , piece_id=TQVPL6ZA6LHKAJVL6XWSYASZD3MCKKV3553CRFD72XWTVWY62STQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.069183369 (524h17m24.531275862s) [noise=true , address=145.79.227.5:4222 , piece_id=C2JSPN2QVSGSXG6ZW3ZMOFX5YFNN65XY7E62N5OEHOOMK3NSO4KQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.070534782 (524h17m24.532627265s) [noise=true , address=77.90.31.11:4555 , piece_id=U5RYZSD643KSUI4XLL26EAN6FQRPMG62GQMMCQRPEEG2XKSNBIZQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.072356982 (524h17m24.534449466s) [noise=true , address=145.79.227.11:4730 , piece_id=VO2DSM5LT43QAFPWOY3W3PVEJ44GZJWTCWG5QRQQFB363DFVFZDA, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.073961167 (524h17m24.53605364s) [noise=true , address=45.91.116.4:4940 , piece_id=3MCSUDYN73NIVJTGKPOVY6N3MALB3GBXI7I5NAX4HXPDHVMQ52BA, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.075716960 (524h17m24.537809433s) [noise=true , address=145.79.227.16:4910 , piece_id=BIHEJDH7RPI4UCPUNV453QHSEVCLA54PSNNQB2II37XZDRXFZNWQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.079266419 (524h17m24.541358902s) [noise=true , address=145.79.227.14:4838 , piece_id=6K6A4CNEHGB3X6DEBLF5KHLUT4GE6BQBW6IE3RHBML2NWPN255AA, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.080943032 (524h17m24.543035505s) [noise=true , address=45.91.116.12:4226 , piece_id=DRXRGO7OXEKVS5ACL6FC7RKLGBBD6KNI7X6E5DM5D5VV5PEKN7FQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.081340479 (524h17m24.543432962s) [noise=true , address=45.91.116.8:4045 , piece_id=EDXBNLKKRZMT6MKK2KOIZM7DDKA3IHB6UMCXX4YKZF7JY2T3S5CQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.083616223 (524h17m24.545708706s) [noise=true , address=145.79.227.14:4840 , piece_id=B4ZVPRMVH7Y3TCWCLPO7DWRVMJL7VSI73CIT4PEHCMTIEURQKB2Q, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.083633946 (524h17m24.545726429s) [noise=true , address=45.91.116.14:4189 , piece_id=UMQRQYPZ7SWUCO2N7JDCEVI5U5P36FDL3SUM6HIYJVWVOZTWNXQQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.085493247 (524h17m24.54758573s) [noise=true , address=145.79.227.17:4969 , piece_id=R5HIP33QW5HLICARFWDUOW7I75YTDHZTJCIML5X52VKHLPL3J4YQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:40.727263388 (524h17m26.189355871s) [upload=461218]: Next returned nil segment Oct 15 09:48:40.727310076 (524h17m26.189402549s) [upload=461218]: Waiting for error group managing segments... Oct 15 09:48:41.004514306 (524h17m26.466606789s) [noise=true , address=145.79.227.7:4294 , piece_id=H4ZNVIXORNO4MTXOVYDQZI4IJE4VXHZWC6JI35YKOR3UESSXJNTQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.006116557 (524h17m26.46820904s) [noise=true , address=145.79.227.5:4222 , piece_id=C2JSPN2QVSGSXG6ZW3ZMOFX5YFNN65XY7E62N5OEHOOMK3NSO4KQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.012963184 (524h17m26.475055667s) [noise=true , address=145.79.227.16:4910 , piece_id=BIHEJDH7RPI4UCPUNV453QHSEVCLA54PSNNQB2II37XZDRXFZNWQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.012994684 (524h17m26.475087167s) [noise=true , address=145.79.227.14:4840 , piece_id=B4ZVPRMVH7Y3TCWCLPO7DWRVMJL7VSI73CIT4PEHCMTIEURQKB2Q, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.016649956 (524h17m26.478742439s) [noise=true , address=145.79.227.11:4730 , piece_id=VO2DSM5LT43QAFPWOY3W3PVEJ44GZJWTCWG5QRQQFB363DFVFZDA, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.019248864 (524h17m26.481341347s) [noise=true , address=145.79.227.17:4969 , piece_id=R5HIP33QW5HLICARFWDUOW7I75YTDHZTJCIML5X52VKHLPL3J4YQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.075991585 (524h17m26.538084068s) [noise=true , address=145.79.227.4:4187 , piece_id=TQVPL6ZA6LHKAJVL6XWSYASZD3MCKKV3553CRFD72XWTVWY62STQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.118655146 (524h17m26.580747619s) [noise=true , address=145.79.227.14:4838 , piece_id=6K6A4CNEHGB3X6DEBLF5KHLUT4GE6BQBW6IE3RHBML2NWPN255AA, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.285934795 (524h17m26.748027278s) [noise=true , address=77.90.31.11:4555 , piece_id=U5RYZSD643KSUI4XLL26EAN6FQRPMG62GQMMCQRPEEG2XKSNBIZQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.744525465 (524h17m27.206617948s) [noise=true , address=45.91.116.9:4262 , piece_id=X53JZGRLZLX53UZLLCXXUQCFSQD2FO76X74OM45G26HNDU55UXXQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:42.218961989 (524h17m27.681054462s) [noise=true , address=45.91.116.11:4117 , piece_id=VH7ZJKSZ2NUXQJZE2IKNWP5PAL3TN2NNHD3ATZUQHGKKYMSSM3LQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:42.313638675 (524h17m27.775731158s) [noise=true , address=45.91.116.14:4189 , piece_id=UMQRQYPZ7SWUCO2N7JDCEVI5U5P36FDL3SUM6HIYJVWVOZTWNXQQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:42.434283559 (524h17m27.896376053s) [noise=true , address=45.91.116.8:4045 , piece_id=EDXBNLKKRZMT6MKK2KOIZM7DDKA3IHB6UMCXX4YKZF7JY2T3S5CQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:42.434321832 (524h17m27.896414305s) [seg_pos={0 0}, upload=461218]: Segment reached optimal threshold of 13 pieces. Oct 15 09:48:42.434605041 (524h17m27.896697534s) [noise=true , address=45.91.116.5:4299 , piece_id=4ARMXXE5LBRE2OZQEIITELKPTW4TISZWTD6XZGI4UJV7UCLUXAEQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX): protocol: expected piece hash; context canceled; EOF Oct 15 09:48:42.434666138 (524h17m27.896758621s) [noise=true , address=45.91.116.5:4299 , piece_id=4ARMXXE5LBRE2OZQEIITELKPTW4TISZWTD6XZGI4UJV7UCLUXAEQ, seg_pos={0 0}, upload=461218]: Long tail cancellation due to optimal threshold Oct 15 09:48:42.434692849 (524h17m27.896785432s) [noise=true , address=45.91.116.12:4226 , piece_id=DRXRGO7OXEKVS5ACL6FC7RKLGBBD6KNI7X6E5DM5D5VV5PEKN7FQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1): protocol: expected piece hash; context canceled; EOF Oct 15 09:48:42.434706184 (524h17m27.896798657s) [noise=true , address=45.91.116.12:4226 , piece_id=DRXRGO7OXEKVS5ACL6FC7RKLGBBD6KNI7X6E5DM5D5VV5PEKN7FQ, seg_pos={0 0}, upload=461218]: Long tail cancellation due to optimal threshold Oct 15 09:48:42.434718318 (524h17m27.896810911s) [noise=true , address=45.91.116.4:4940 , piece_id=3MCSUDYN73NIVJTGKPOVY6N3MALB3GBXI7I5NAX4HXPDHVMQ52BA, seg_pos={0 0}, upload=461218]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe): protocol: expected piece hash; context canceled; EOF Oct 15 09:48:42.434728447 (524h17m27.89682093s) [noise=true , address=45.91.116.4:4940 , piece_id=3MCSUDYN73NIVJTGKPOVY6N3MALB3GBXI7I5NAX4HXPDHVMQ52BA, seg_pos={0 0}, upload=461218]: Long tail cancellation due to optimal threshold Oct 15 09:48:42.434760458 (524h17m27.896852931s) [seg_pos={0 0}, upload=461218]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 15 09:48:42.434866900 (524h17m27.896959384s) [upload=461218]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 15 09:48:42.434873884 (524h17m27.896966367s) [upload=461218]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 15 09:48:42.760633614 (524h17m28.222726087s) [upload=461218]: Done uploading segments... Oct 15 09:48:42.760655435 (524h17m28.222747918s) [upload=461218]: Upload finished. err: <nil> Duration: 6.162962178s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.....................X...X..................................] duration: 6.114591008s FAIL (2025-10-15 10:12:52 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, Piece ID: VH7ZJKSZ2NUXQJZE2IKNWP5PAL3TN2NNHD3ATZUQHGKKYMSSM3LQ) context deadline exceeded; piecestore: (Node ID: 1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N, Piece ID: UMQRQYPZ7SWUCO2N7JDCEVI5U5P36FDL3SUM6HIYJVWVOZTWNXQQ) context deadline exceeded ======LOGS====== Oct 15 09:48:38.801058027 (524h17m24.26315051s) [upload=461218]: Starting upload Oct 15 09:48:38.801138119 (524h17m24.263230592s) [upload=461218]: Done starting upload Oct 15 09:48:38.801288545 (524h17m24.263381028s) [upload=461218]: Uploading segments... Oct 15 09:48:38.801442509 (524h17m24.263535002s) [upload=461218]: Got next segment. Inline: false Oct 15 09:48:38.801486663 (524h17m24.263579136s) [upload=461218]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 15 09:48:38.801493245 (524h17m24.263585728s) [upload=461218]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 15 09:48:39.066131918 (524h17m24.528224401s) [seg_pos={0 0}, upload=461218]: Begin upload segment... Oct 15 09:48:39.066205549 (524h17m24.528298022s) [seg_pos={0 0}, upload=461218]: Done begin upload segment. Oct 15 09:48:39.066243090 (524h17m24.528335573s) [noise=true , address=45.91.116.5:4299 , piece_id=4ARMXXE5LBRE2OZQEIITELKPTW4TISZWTD6XZGI4UJV7UCLUXAEQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.066254431 (524h17m24.528346915s) [noise=true , address=45.91.116.11:4117 , piece_id=VH7ZJKSZ2NUXQJZE2IKNWP5PAL3TN2NNHD3ATZUQHGKKYMSSM3LQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.066874873 (524h17m24.528967356s) [noise=true , address=145.79.227.7:4294 , piece_id=H4ZNVIXORNO4MTXOVYDQZI4IJE4VXHZWC6JI35YKOR3UESSXJNTQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.067301676 (524h17m24.529394159s) [noise=true , address=45.91.116.9:4262 , piece_id=X53JZGRLZLX53UZLLCXXUQCFSQD2FO76X74OM45G26HNDU55UXXQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.067705735 (524h17m24.529798218s) [noise=true , address=145.79.227.4:4187 , piece_id=TQVPL6ZA6LHKAJVL6XWSYASZD3MCKKV3553CRFD72XWTVWY62STQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.069183369 (524h17m24.531275862s) [noise=true , address=145.79.227.5:4222 , piece_id=C2JSPN2QVSGSXG6ZW3ZMOFX5YFNN65XY7E62N5OEHOOMK3NSO4KQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.070534782 (524h17m24.532627265s) [noise=true , address=77.90.31.11:4555 , piece_id=U5RYZSD643KSUI4XLL26EAN6FQRPMG62GQMMCQRPEEG2XKSNBIZQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.072356982 (524h17m24.534449466s) [noise=true , address=145.79.227.11:4730 , piece_id=VO2DSM5LT43QAFPWOY3W3PVEJ44GZJWTCWG5QRQQFB363DFVFZDA, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.073961167 (524h17m24.53605364s) [noise=true , address=45.91.116.4:4940 , piece_id=3MCSUDYN73NIVJTGKPOVY6N3MALB3GBXI7I5NAX4HXPDHVMQ52BA, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.075716960 (524h17m24.537809433s) [noise=true , address=145.79.227.16:4910 , piece_id=BIHEJDH7RPI4UCPUNV453QHSEVCLA54PSNNQB2II37XZDRXFZNWQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.079266419 (524h17m24.541358902s) [noise=true , address=145.79.227.14:4838 , piece_id=6K6A4CNEHGB3X6DEBLF5KHLUT4GE6BQBW6IE3RHBML2NWPN255AA, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.080943032 (524h17m24.543035505s) [noise=true , address=45.91.116.12:4226 , piece_id=DRXRGO7OXEKVS5ACL6FC7RKLGBBD6KNI7X6E5DM5D5VV5PEKN7FQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.081340479 (524h17m24.543432962s) [noise=true , address=45.91.116.8:4045 , piece_id=EDXBNLKKRZMT6MKK2KOIZM7DDKA3IHB6UMCXX4YKZF7JY2T3S5CQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.083616223 (524h17m24.545708706s) [noise=true , address=145.79.227.14:4840 , piece_id=B4ZVPRMVH7Y3TCWCLPO7DWRVMJL7VSI73CIT4PEHCMTIEURQKB2Q, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.083633946 (524h17m24.545726429s) [noise=true , address=45.91.116.14:4189 , piece_id=UMQRQYPZ7SWUCO2N7JDCEVI5U5P36FDL3SUM6HIYJVWVOZTWNXQQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.085493247 (524h17m24.54758573s) [noise=true , address=145.79.227.17:4969 , piece_id=R5HIP33QW5HLICARFWDUOW7I75YTDHZTJCIML5X52VKHLPL3J4YQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:40.727263388 (524h17m26.189355871s) [upload=461218]: Next returned nil segment Oct 15 09:48:40.727310076 (524h17m26.189402549s) [upload=461218]: Waiting for error group managing segments... Oct 15 09:48:41.004514306 (524h17m26.466606789s) [noise=true , address=145.79.227.7:4294 , piece_id=H4ZNVIXORNO4MTXOVYDQZI4IJE4VXHZWC6JI35YKOR3UESSXJNTQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.006116557 (524h17m26.46820904s) [noise=true , address=145.79.227.5:4222 , piece_id=C2JSPN2QVSGSXG6ZW3ZMOFX5YFNN65XY7E62N5OEHOOMK3NSO4KQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.012963184 (524h17m26.475055667s) [noise=true , address=145.79.227.16:4910 , piece_id=BIHEJDH7RPI4UCPUNV453QHSEVCLA54PSNNQB2II37XZDRXFZNWQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.012994684 (524h17m26.475087167s) [noise=true , address=145.79.227.14:4840 , piece_id=B4ZVPRMVH7Y3TCWCLPO7DWRVMJL7VSI73CIT4PEHCMTIEURQKB2Q, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.016649956 (524h17m26.478742439s) [noise=true , address=145.79.227.11:4730 , piece_id=VO2DSM5LT43QAFPWOY3W3PVEJ44GZJWTCWG5QRQQFB363DFVFZDA, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.019248864 (524h17m26.481341347s) [noise=true , address=145.79.227.17:4969 , piece_id=R5HIP33QW5HLICARFWDUOW7I75YTDHZTJCIML5X52VKHLPL3J4YQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.075991585 (524h17m26.538084068s) [noise=true , address=145.79.227.4:4187 , piece_id=TQVPL6ZA6LHKAJVL6XWSYASZD3MCKKV3553CRFD72XWTVWY62STQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.118655146 (524h17m26.580747619s) [noise=true , address=145.79.227.14:4838 , piece_id=6K6A4CNEHGB3X6DEBLF5KHLUT4GE6BQBW6IE3RHBML2NWPN255AA, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.285934795 (524h17m26.748027278s) [noise=true , address=77.90.31.11:4555 , piece_id=U5RYZSD643KSUI4XLL26EAN6FQRPMG62GQMMCQRPEEG2XKSNBIZQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.744525465 (524h17m27.206617948s) [noise=true , address=45.91.116.9:4262 , piece_id=X53JZGRLZLX53UZLLCXXUQCFSQD2FO76X74OM45G26HNDU55UXXQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:42.218961989 (524h17m27.681054462s) [noise=true , address=45.91.116.11:4117 , piece_id=VH7ZJKSZ2NUXQJZE2IKNWP5PAL3TN2NNHD3ATZUQHGKKYMSSM3LQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:42.313638675 (524h17m27.775731158s) [noise=true , address=45.91.116.14:4189 , piece_id=UMQRQYPZ7SWUCO2N7JDCEVI5U5P36FDL3SUM6HIYJVWVOZTWNXQQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:42.434283559 (524h17m27.896376053s) [noise=true , address=45.91.116.8:4045 , piece_id=EDXBNLKKRZMT6MKK2KOIZM7DDKA3IHB6UMCXX4YKZF7JY2T3S5CQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:42.434321832 (524h17m27.896414305s) [seg_pos={0 0}, upload=461218]: Segment reached optimal threshold of 13 pieces. Oct 15 09:48:42.434605041 (524h17m27.896697534s) [noise=true , address=45.91.116.5:4299 , piece_id=4ARMXXE5LBRE2OZQEIITELKPTW4TISZWTD6XZGI4UJV7UCLUXAEQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX): protocol: expected piece hash; context canceled; EOF Oct 15 09:48:42.434666138 (524h17m27.896758621s) [noise=true , address=45.91.116.5:4299 , piece_id=4ARMXXE5LBRE2OZQEIITELKPTW4TISZWTD6XZGI4UJV7UCLUXAEQ, seg_pos={0 0}, upload=461218]: Long tail cancellation due to optimal threshold Oct 15 09:48:42.434692849 (524h17m27.896785432s) [noise=true , address=45.91.116.12:4226 , piece_id=DRXRGO7OXEKVS5ACL6FC7RKLGBBD6KNI7X6E5DM5D5VV5PEKN7FQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1): protocol: expected piece hash; context canceled; EOF Oct 15 09:48:42.434706184 (524h17m27.896798657s) [noise=true , address=45.91.116.12:4226 , piece_id=DRXRGO7OXEKVS5ACL6FC7RKLGBBD6KNI7X6E5DM5D5VV5PEKN7FQ, seg_pos={0 0}, upload=461218]: Long tail cancellation due to optimal threshold Oct 15 09:48:42.434718318 (524h17m27.896810911s) [noise=true , address=45.91.116.4:4940 , piece_id=3MCSUDYN73NIVJTGKPOVY6N3MALB3GBXI7I5NAX4HXPDHVMQ52BA, seg_pos={0 0}, upload=461218]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe): protocol: expected piece hash; context canceled; EOF Oct 15 09:48:42.434728447 (524h17m27.89682093s) [noise=true , address=45.91.116.4:4940 , piece_id=3MCSUDYN73NIVJTGKPOVY6N3MALB3GBXI7I5NAX4HXPDHVMQ52BA, seg_pos={0 0}, upload=461218]: Long tail cancellation due to optimal threshold Oct 15 09:48:42.434760458 (524h17m27.896852931s) [seg_pos={0 0}, upload=461218]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 15 09:48:42.434866900 (524h17m27.896959384s) [upload=461218]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 15 09:48:42.434873884 (524h17m27.896966367s) [upload=461218]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 15 09:48:42.760633614 (524h17m28.222726087s) [upload=461218]: Done uploading segments... Oct 15 09:48:42.760655435 (524h17m28.222747918s) [upload=461218]: Upload finished. err: <nil> Duration: 6.114625665s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [......................X...X.................................] duration: 5.965682934s FAIL (2025-10-15 10:12:37 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, Piece ID: VH7ZJKSZ2NUXQJZE2IKNWP5PAL3TN2NNHD3ATZUQHGKKYMSSM3LQ) context deadline exceeded; piecestore: (Node ID: 1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N, Piece ID: UMQRQYPZ7SWUCO2N7JDCEVI5U5P36FDL3SUM6HIYJVWVOZTWNXQQ) context deadline exceeded ======LOGS====== Oct 15 09:48:38.801058027 (524h17m24.26315051s) [upload=461218]: Starting upload Oct 15 09:48:38.801138119 (524h17m24.263230592s) [upload=461218]: Done starting upload Oct 15 09:48:38.801288545 (524h17m24.263381028s) [upload=461218]: Uploading segments... Oct 15 09:48:38.801442509 (524h17m24.263535002s) [upload=461218]: Got next segment. Inline: false Oct 15 09:48:38.801486663 (524h17m24.263579136s) [upload=461218]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 15 09:48:38.801493245 (524h17m24.263585728s) [upload=461218]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 15 09:48:39.066131918 (524h17m24.528224401s) [seg_pos={0 0}, upload=461218]: Begin upload segment... Oct 15 09:48:39.066205549 (524h17m24.528298022s) [seg_pos={0 0}, upload=461218]: Done begin upload segment. Oct 15 09:48:39.066243090 (524h17m24.528335573s) [noise=true , address=45.91.116.5:4299 , piece_id=4ARMXXE5LBRE2OZQEIITELKPTW4TISZWTD6XZGI4UJV7UCLUXAEQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.066254431 (524h17m24.528346915s) [noise=true , address=45.91.116.11:4117 , piece_id=VH7ZJKSZ2NUXQJZE2IKNWP5PAL3TN2NNHD3ATZUQHGKKYMSSM3LQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.066874873 (524h17m24.528967356s) [noise=true , address=145.79.227.7:4294 , piece_id=H4ZNVIXORNO4MTXOVYDQZI4IJE4VXHZWC6JI35YKOR3UESSXJNTQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.067301676 (524h17m24.529394159s) [noise=true , address=45.91.116.9:4262 , piece_id=X53JZGRLZLX53UZLLCXXUQCFSQD2FO76X74OM45G26HNDU55UXXQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.067705735 (524h17m24.529798218s) [noise=true , address=145.79.227.4:4187 , piece_id=TQVPL6ZA6LHKAJVL6XWSYASZD3MCKKV3553CRFD72XWTVWY62STQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.069183369 (524h17m24.531275862s) [noise=true , address=145.79.227.5:4222 , piece_id=C2JSPN2QVSGSXG6ZW3ZMOFX5YFNN65XY7E62N5OEHOOMK3NSO4KQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.070534782 (524h17m24.532627265s) [noise=true , address=77.90.31.11:4555 , piece_id=U5RYZSD643KSUI4XLL26EAN6FQRPMG62GQMMCQRPEEG2XKSNBIZQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.072356982 (524h17m24.534449466s) [noise=true , address=145.79.227.11:4730 , piece_id=VO2DSM5LT43QAFPWOY3W3PVEJ44GZJWTCWG5QRQQFB363DFVFZDA, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.073961167 (524h17m24.53605364s) [noise=true , address=45.91.116.4:4940 , piece_id=3MCSUDYN73NIVJTGKPOVY6N3MALB3GBXI7I5NAX4HXPDHVMQ52BA, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.075716960 (524h17m24.537809433s) [noise=true , address=145.79.227.16:4910 , piece_id=BIHEJDH7RPI4UCPUNV453QHSEVCLA54PSNNQB2II37XZDRXFZNWQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.079266419 (524h17m24.541358902s) [noise=true , address=145.79.227.14:4838 , piece_id=6K6A4CNEHGB3X6DEBLF5KHLUT4GE6BQBW6IE3RHBML2NWPN255AA, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.080943032 (524h17m24.543035505s) [noise=true , address=45.91.116.12:4226 , piece_id=DRXRGO7OXEKVS5ACL6FC7RKLGBBD6KNI7X6E5DM5D5VV5PEKN7FQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.081340479 (524h17m24.543432962s) [noise=true , address=45.91.116.8:4045 , piece_id=EDXBNLKKRZMT6MKK2KOIZM7DDKA3IHB6UMCXX4YKZF7JY2T3S5CQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.083616223 (524h17m24.545708706s) [noise=true , address=145.79.227.14:4840 , piece_id=B4ZVPRMVH7Y3TCWCLPO7DWRVMJL7VSI73CIT4PEHCMTIEURQKB2Q, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.083633946 (524h17m24.545726429s) [noise=true , address=45.91.116.14:4189 , piece_id=UMQRQYPZ7SWUCO2N7JDCEVI5U5P36FDL3SUM6HIYJVWVOZTWNXQQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.085493247 (524h17m24.54758573s) [noise=true , address=145.79.227.17:4969 , piece_id=R5HIP33QW5HLICARFWDUOW7I75YTDHZTJCIML5X52VKHLPL3J4YQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:40.727263388 (524h17m26.189355871s) [upload=461218]: Next returned nil segment Oct 15 09:48:40.727310076 (524h17m26.189402549s) [upload=461218]: Waiting for error group managing segments... Oct 15 09:48:41.004514306 (524h17m26.466606789s) [noise=true , address=145.79.227.7:4294 , piece_id=H4ZNVIXORNO4MTXOVYDQZI4IJE4VXHZWC6JI35YKOR3UESSXJNTQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.006116557 (524h17m26.46820904s) [noise=true , address=145.79.227.5:4222 , piece_id=C2JSPN2QVSGSXG6ZW3ZMOFX5YFNN65XY7E62N5OEHOOMK3NSO4KQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.012963184 (524h17m26.475055667s) [noise=true , address=145.79.227.16:4910 , piece_id=BIHEJDH7RPI4UCPUNV453QHSEVCLA54PSNNQB2II37XZDRXFZNWQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.012994684 (524h17m26.475087167s) [noise=true , address=145.79.227.14:4840 , piece_id=B4ZVPRMVH7Y3TCWCLPO7DWRVMJL7VSI73CIT4PEHCMTIEURQKB2Q, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.016649956 (524h17m26.478742439s) [noise=true , address=145.79.227.11:4730 , piece_id=VO2DSM5LT43QAFPWOY3W3PVEJ44GZJWTCWG5QRQQFB363DFVFZDA, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.019248864 (524h17m26.481341347s) [noise=true , address=145.79.227.17:4969 , piece_id=R5HIP33QW5HLICARFWDUOW7I75YTDHZTJCIML5X52VKHLPL3J4YQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.075991585 (524h17m26.538084068s) [noise=true , address=145.79.227.4:4187 , piece_id=TQVPL6ZA6LHKAJVL6XWSYASZD3MCKKV3553CRFD72XWTVWY62STQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.118655146 (524h17m26.580747619s) [noise=true , address=145.79.227.14:4838 , piece_id=6K6A4CNEHGB3X6DEBLF5KHLUT4GE6BQBW6IE3RHBML2NWPN255AA, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.285934795 (524h17m26.748027278s) [noise=true , address=77.90.31.11:4555 , piece_id=U5RYZSD643KSUI4XLL26EAN6FQRPMG62GQMMCQRPEEG2XKSNBIZQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.744525465 (524h17m27.206617948s) [noise=true , address=45.91.116.9:4262 , piece_id=X53JZGRLZLX53UZLLCXXUQCFSQD2FO76X74OM45G26HNDU55UXXQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:42.218961989 (524h17m27.681054462s) [noise=true , address=45.91.116.11:4117 , piece_id=VH7ZJKSZ2NUXQJZE2IKNWP5PAL3TN2NNHD3ATZUQHGKKYMSSM3LQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:42.313638675 (524h17m27.775731158s) [noise=true , address=45.91.116.14:4189 , piece_id=UMQRQYPZ7SWUCO2N7JDCEVI5U5P36FDL3SUM6HIYJVWVOZTWNXQQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:42.434283559 (524h17m27.896376053s) [noise=true , address=45.91.116.8:4045 , piece_id=EDXBNLKKRZMT6MKK2KOIZM7DDKA3IHB6UMCXX4YKZF7JY2T3S5CQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:42.434321832 (524h17m27.896414305s) [seg_pos={0 0}, upload=461218]: Segment reached optimal threshold of 13 pieces. Oct 15 09:48:42.434605041 (524h17m27.896697534s) [noise=true , address=45.91.116.5:4299 , piece_id=4ARMXXE5LBRE2OZQEIITELKPTW4TISZWTD6XZGI4UJV7UCLUXAEQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX): protocol: expected piece hash; context canceled; EOF Oct 15 09:48:42.434666138 (524h17m27.896758621s) [noise=true , address=45.91.116.5:4299 , piece_id=4ARMXXE5LBRE2OZQEIITELKPTW4TISZWTD6XZGI4UJV7UCLUXAEQ, seg_pos={0 0}, upload=461218]: Long tail cancellation due to optimal threshold Oct 15 09:48:42.434692849 (524h17m27.896785432s) [noise=true , address=45.91.116.12:4226 , piece_id=DRXRGO7OXEKVS5ACL6FC7RKLGBBD6KNI7X6E5DM5D5VV5PEKN7FQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1): protocol: expected piece hash; context canceled; EOF Oct 15 09:48:42.434706184 (524h17m27.896798657s) [noise=true , address=45.91.116.12:4226 , piece_id=DRXRGO7OXEKVS5ACL6FC7RKLGBBD6KNI7X6E5DM5D5VV5PEKN7FQ, seg_pos={0 0}, upload=461218]: Long tail cancellation due to optimal threshold Oct 15 09:48:42.434718318 (524h17m27.896810911s) [noise=true , address=45.91.116.4:4940 , piece_id=3MCSUDYN73NIVJTGKPOVY6N3MALB3GBXI7I5NAX4HXPDHVMQ52BA, seg_pos={0 0}, upload=461218]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe): protocol: expected piece hash; context canceled; EOF Oct 15 09:48:42.434728447 (524h17m27.89682093s) [noise=true , address=45.91.116.4:4940 , piece_id=3MCSUDYN73NIVJTGKPOVY6N3MALB3GBXI7I5NAX4HXPDHVMQ52BA, seg_pos={0 0}, upload=461218]: Long tail cancellation due to optimal threshold Oct 15 09:48:42.434760458 (524h17m27.896852931s) [seg_pos={0 0}, upload=461218]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 15 09:48:42.434866900 (524h17m27.896959384s) [upload=461218]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 15 09:48:42.434873884 (524h17m27.896966367s) [upload=461218]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 15 09:48:42.760633614 (524h17m28.222726087s) [upload=461218]: Done uploading segments... Oct 15 09:48:42.760655435 (524h17m28.222747918s) [upload=461218]: Upload finished. err: <nil> Duration: 5.965711719s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.......................X...X................................] duration: 6.008161032s FAIL (2025-10-15 10:11:41 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, Piece ID: VH7ZJKSZ2NUXQJZE2IKNWP5PAL3TN2NNHD3ATZUQHGKKYMSSM3LQ) context deadline exceeded; piecestore: (Node ID: 1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N, Piece ID: UMQRQYPZ7SWUCO2N7JDCEVI5U5P36FDL3SUM6HIYJVWVOZTWNXQQ) context deadline exceeded ======LOGS====== Oct 15 09:48:38.801058027 (524h17m24.26315051s) [upload=461218]: Starting upload Oct 15 09:48:38.801138119 (524h17m24.263230592s) [upload=461218]: Done starting upload Oct 15 09:48:38.801288545 (524h17m24.263381028s) [upload=461218]: Uploading segments... Oct 15 09:48:38.801442509 (524h17m24.263535002s) [upload=461218]: Got next segment. Inline: false Oct 15 09:48:38.801486663 (524h17m24.263579136s) [upload=461218]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 15 09:48:38.801493245 (524h17m24.263585728s) [upload=461218]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 15 09:48:39.066131918 (524h17m24.528224401s) [seg_pos={0 0}, upload=461218]: Begin upload segment... Oct 15 09:48:39.066205549 (524h17m24.528298022s) [seg_pos={0 0}, upload=461218]: Done begin upload segment. Oct 15 09:48:39.066243090 (524h17m24.528335573s) [noise=true , address=45.91.116.5:4299 , piece_id=4ARMXXE5LBRE2OZQEIITELKPTW4TISZWTD6XZGI4UJV7UCLUXAEQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.066254431 (524h17m24.528346915s) [noise=true , address=45.91.116.11:4117 , piece_id=VH7ZJKSZ2NUXQJZE2IKNWP5PAL3TN2NNHD3ATZUQHGKKYMSSM3LQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.066874873 (524h17m24.528967356s) [noise=true , address=145.79.227.7:4294 , piece_id=H4ZNVIXORNO4MTXOVYDQZI4IJE4VXHZWC6JI35YKOR3UESSXJNTQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.067301676 (524h17m24.529394159s) [noise=true , address=45.91.116.9:4262 , piece_id=X53JZGRLZLX53UZLLCXXUQCFSQD2FO76X74OM45G26HNDU55UXXQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.067705735 (524h17m24.529798218s) [noise=true , address=145.79.227.4:4187 , piece_id=TQVPL6ZA6LHKAJVL6XWSYASZD3MCKKV3553CRFD72XWTVWY62STQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.069183369 (524h17m24.531275862s) [noise=true , address=145.79.227.5:4222 , piece_id=C2JSPN2QVSGSXG6ZW3ZMOFX5YFNN65XY7E62N5OEHOOMK3NSO4KQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.070534782 (524h17m24.532627265s) [noise=true , address=77.90.31.11:4555 , piece_id=U5RYZSD643KSUI4XLL26EAN6FQRPMG62GQMMCQRPEEG2XKSNBIZQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.072356982 (524h17m24.534449466s) [noise=true , address=145.79.227.11:4730 , piece_id=VO2DSM5LT43QAFPWOY3W3PVEJ44GZJWTCWG5QRQQFB363DFVFZDA, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.073961167 (524h17m24.53605364s) [noise=true , address=45.91.116.4:4940 , piece_id=3MCSUDYN73NIVJTGKPOVY6N3MALB3GBXI7I5NAX4HXPDHVMQ52BA, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.075716960 (524h17m24.537809433s) [noise=true , address=145.79.227.16:4910 , piece_id=BIHEJDH7RPI4UCPUNV453QHSEVCLA54PSNNQB2II37XZDRXFZNWQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.079266419 (524h17m24.541358902s) [noise=true , address=145.79.227.14:4838 , piece_id=6K6A4CNEHGB3X6DEBLF5KHLUT4GE6BQBW6IE3RHBML2NWPN255AA, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.080943032 (524h17m24.543035505s) [noise=true , address=45.91.116.12:4226 , piece_id=DRXRGO7OXEKVS5ACL6FC7RKLGBBD6KNI7X6E5DM5D5VV5PEKN7FQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.081340479 (524h17m24.543432962s) [noise=true , address=45.91.116.8:4045 , piece_id=EDXBNLKKRZMT6MKK2KOIZM7DDKA3IHB6UMCXX4YKZF7JY2T3S5CQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.083616223 (524h17m24.545708706s) [noise=true , address=145.79.227.14:4840 , piece_id=B4ZVPRMVH7Y3TCWCLPO7DWRVMJL7VSI73CIT4PEHCMTIEURQKB2Q, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.083633946 (524h17m24.545726429s) [noise=true , address=45.91.116.14:4189 , piece_id=UMQRQYPZ7SWUCO2N7JDCEVI5U5P36FDL3SUM6HIYJVWVOZTWNXQQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.085493247 (524h17m24.54758573s) [noise=true , address=145.79.227.17:4969 , piece_id=R5HIP33QW5HLICARFWDUOW7I75YTDHZTJCIML5X52VKHLPL3J4YQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:40.727263388 (524h17m26.189355871s) [upload=461218]: Next returned nil segment Oct 15 09:48:40.727310076 (524h17m26.189402549s) [upload=461218]: Waiting for error group managing segments... Oct 15 09:48:41.004514306 (524h17m26.466606789s) [noise=true , address=145.79.227.7:4294 , piece_id=H4ZNVIXORNO4MTXOVYDQZI4IJE4VXHZWC6JI35YKOR3UESSXJNTQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.006116557 (524h17m26.46820904s) [noise=true , address=145.79.227.5:4222 , piece_id=C2JSPN2QVSGSXG6ZW3ZMOFX5YFNN65XY7E62N5OEHOOMK3NSO4KQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.012963184 (524h17m26.475055667s) [noise=true , address=145.79.227.16:4910 , piece_id=BIHEJDH7RPI4UCPUNV453QHSEVCLA54PSNNQB2II37XZDRXFZNWQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.012994684 (524h17m26.475087167s) [noise=true , address=145.79.227.14:4840 , piece_id=B4ZVPRMVH7Y3TCWCLPO7DWRVMJL7VSI73CIT4PEHCMTIEURQKB2Q, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.016649956 (524h17m26.478742439s) [noise=true , address=145.79.227.11:4730 , piece_id=VO2DSM5LT43QAFPWOY3W3PVEJ44GZJWTCWG5QRQQFB363DFVFZDA, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.019248864 (524h17m26.481341347s) [noise=true , address=145.79.227.17:4969 , piece_id=R5HIP33QW5HLICARFWDUOW7I75YTDHZTJCIML5X52VKHLPL3J4YQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.075991585 (524h17m26.538084068s) [noise=true , address=145.79.227.4:4187 , piece_id=TQVPL6ZA6LHKAJVL6XWSYASZD3MCKKV3553CRFD72XWTVWY62STQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.118655146 (524h17m26.580747619s) [noise=true , address=145.79.227.14:4838 , piece_id=6K6A4CNEHGB3X6DEBLF5KHLUT4GE6BQBW6IE3RHBML2NWPN255AA, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.285934795 (524h17m26.748027278s) [noise=true , address=77.90.31.11:4555 , piece_id=U5RYZSD643KSUI4XLL26EAN6FQRPMG62GQMMCQRPEEG2XKSNBIZQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.744525465 (524h17m27.206617948s) [noise=true , address=45.91.116.9:4262 , piece_id=X53JZGRLZLX53UZLLCXXUQCFSQD2FO76X74OM45G26HNDU55UXXQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:42.218961989 (524h17m27.681054462s) [noise=true , address=45.91.116.11:4117 , piece_id=VH7ZJKSZ2NUXQJZE2IKNWP5PAL3TN2NNHD3ATZUQHGKKYMSSM3LQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:42.313638675 (524h17m27.775731158s) [noise=true , address=45.91.116.14:4189 , piece_id=UMQRQYPZ7SWUCO2N7JDCEVI5U5P36FDL3SUM6HIYJVWVOZTWNXQQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:42.434283559 (524h17m27.896376053s) [noise=true , address=45.91.116.8:4045 , piece_id=EDXBNLKKRZMT6MKK2KOIZM7DDKA3IHB6UMCXX4YKZF7JY2T3S5CQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:42.434321832 (524h17m27.896414305s) [seg_pos={0 0}, upload=461218]: Segment reached optimal threshold of 13 pieces. Oct 15 09:48:42.434605041 (524h17m27.896697534s) [noise=true , address=45.91.116.5:4299 , piece_id=4ARMXXE5LBRE2OZQEIITELKPTW4TISZWTD6XZGI4UJV7UCLUXAEQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX): protocol: expected piece hash; context canceled; EOF Oct 15 09:48:42.434666138 (524h17m27.896758621s) [noise=true , address=45.91.116.5:4299 , piece_id=4ARMXXE5LBRE2OZQEIITELKPTW4TISZWTD6XZGI4UJV7UCLUXAEQ, seg_pos={0 0}, upload=461218]: Long tail cancellation due to optimal threshold Oct 15 09:48:42.434692849 (524h17m27.896785432s) [noise=true , address=45.91.116.12:4226 , piece_id=DRXRGO7OXEKVS5ACL6FC7RKLGBBD6KNI7X6E5DM5D5VV5PEKN7FQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1): protocol: expected piece hash; context canceled; EOF Oct 15 09:48:42.434706184 (524h17m27.896798657s) [noise=true , address=45.91.116.12:4226 , piece_id=DRXRGO7OXEKVS5ACL6FC7RKLGBBD6KNI7X6E5DM5D5VV5PEKN7FQ, seg_pos={0 0}, upload=461218]: Long tail cancellation due to optimal threshold Oct 15 09:48:42.434718318 (524h17m27.896810911s) [noise=true , address=45.91.116.4:4940 , piece_id=3MCSUDYN73NIVJTGKPOVY6N3MALB3GBXI7I5NAX4HXPDHVMQ52BA, seg_pos={0 0}, upload=461218]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe): protocol: expected piece hash; context canceled; EOF Oct 15 09:48:42.434728447 (524h17m27.89682093s) [noise=true , address=45.91.116.4:4940 , piece_id=3MCSUDYN73NIVJTGKPOVY6N3MALB3GBXI7I5NAX4HXPDHVMQ52BA, seg_pos={0 0}, upload=461218]: Long tail cancellation due to optimal threshold Oct 15 09:48:42.434760458 (524h17m27.896852931s) [seg_pos={0 0}, upload=461218]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 15 09:48:42.434866900 (524h17m27.896959384s) [upload=461218]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 15 09:48:42.434873884 (524h17m27.896966367s) [upload=461218]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 15 09:48:42.760633614 (524h17m28.222726087s) [upload=461218]: Done uploading segments... Oct 15 09:48:42.760655435 (524h17m28.222747918s) [upload=461218]: Upload finished. err: <nil> Duration: 6.008185119s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [........................X...X...............................] duration: 6.329375507s FAIL (2025-10-15 10:10:49 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, Piece ID: VH7ZJKSZ2NUXQJZE2IKNWP5PAL3TN2NNHD3ATZUQHGKKYMSSM3LQ) context deadline exceeded; piecestore: (Node ID: 1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N, Piece ID: UMQRQYPZ7SWUCO2N7JDCEVI5U5P36FDL3SUM6HIYJVWVOZTWNXQQ) context deadline exceeded ======LOGS====== Oct 15 09:48:38.801058027 (524h17m24.26315051s) [upload=461218]: Starting upload Oct 15 09:48:38.801138119 (524h17m24.263230592s) [upload=461218]: Done starting upload Oct 15 09:48:38.801288545 (524h17m24.263381028s) [upload=461218]: Uploading segments... Oct 15 09:48:38.801442509 (524h17m24.263535002s) [upload=461218]: Got next segment. Inline: false Oct 15 09:48:38.801486663 (524h17m24.263579136s) [upload=461218]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 15 09:48:38.801493245 (524h17m24.263585728s) [upload=461218]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 15 09:48:39.066131918 (524h17m24.528224401s) [seg_pos={0 0}, upload=461218]: Begin upload segment... Oct 15 09:48:39.066205549 (524h17m24.528298022s) [seg_pos={0 0}, upload=461218]: Done begin upload segment. Oct 15 09:48:39.066243090 (524h17m24.528335573s) [noise=true , address=45.91.116.5:4299 , piece_id=4ARMXXE5LBRE2OZQEIITELKPTW4TISZWTD6XZGI4UJV7UCLUXAEQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.066254431 (524h17m24.528346915s) [noise=true , address=45.91.116.11:4117 , piece_id=VH7ZJKSZ2NUXQJZE2IKNWP5PAL3TN2NNHD3ATZUQHGKKYMSSM3LQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.066874873 (524h17m24.528967356s) [noise=true , address=145.79.227.7:4294 , piece_id=H4ZNVIXORNO4MTXOVYDQZI4IJE4VXHZWC6JI35YKOR3UESSXJNTQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.067301676 (524h17m24.529394159s) [noise=true , address=45.91.116.9:4262 , piece_id=X53JZGRLZLX53UZLLCXXUQCFSQD2FO76X74OM45G26HNDU55UXXQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.067705735 (524h17m24.529798218s) [noise=true , address=145.79.227.4:4187 , piece_id=TQVPL6ZA6LHKAJVL6XWSYASZD3MCKKV3553CRFD72XWTVWY62STQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.069183369 (524h17m24.531275862s) [noise=true , address=145.79.227.5:4222 , piece_id=C2JSPN2QVSGSXG6ZW3ZMOFX5YFNN65XY7E62N5OEHOOMK3NSO4KQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.070534782 (524h17m24.532627265s) [noise=true , address=77.90.31.11:4555 , piece_id=U5RYZSD643KSUI4XLL26EAN6FQRPMG62GQMMCQRPEEG2XKSNBIZQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.072356982 (524h17m24.534449466s) [noise=true , address=145.79.227.11:4730 , piece_id=VO2DSM5LT43QAFPWOY3W3PVEJ44GZJWTCWG5QRQQFB363DFVFZDA, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.073961167 (524h17m24.53605364s) [noise=true , address=45.91.116.4:4940 , piece_id=3MCSUDYN73NIVJTGKPOVY6N3MALB3GBXI7I5NAX4HXPDHVMQ52BA, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.075716960 (524h17m24.537809433s) [noise=true , address=145.79.227.16:4910 , piece_id=BIHEJDH7RPI4UCPUNV453QHSEVCLA54PSNNQB2II37XZDRXFZNWQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.079266419 (524h17m24.541358902s) [noise=true , address=145.79.227.14:4838 , piece_id=6K6A4CNEHGB3X6DEBLF5KHLUT4GE6BQBW6IE3RHBML2NWPN255AA, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.080943032 (524h17m24.543035505s) [noise=true , address=45.91.116.12:4226 , piece_id=DRXRGO7OXEKVS5ACL6FC7RKLGBBD6KNI7X6E5DM5D5VV5PEKN7FQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.081340479 (524h17m24.543432962s) [noise=true , address=45.91.116.8:4045 , piece_id=EDXBNLKKRZMT6MKK2KOIZM7DDKA3IHB6UMCXX4YKZF7JY2T3S5CQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.083616223 (524h17m24.545708706s) [noise=true , address=145.79.227.14:4840 , piece_id=B4ZVPRMVH7Y3TCWCLPO7DWRVMJL7VSI73CIT4PEHCMTIEURQKB2Q, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.083633946 (524h17m24.545726429s) [noise=true , address=45.91.116.14:4189 , piece_id=UMQRQYPZ7SWUCO2N7JDCEVI5U5P36FDL3SUM6HIYJVWVOZTWNXQQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.085493247 (524h17m24.54758573s) [noise=true , address=145.79.227.17:4969 , piece_id=R5HIP33QW5HLICARFWDUOW7I75YTDHZTJCIML5X52VKHLPL3J4YQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:40.727263388 (524h17m26.189355871s) [upload=461218]: Next returned nil segment Oct 15 09:48:40.727310076 (524h17m26.189402549s) [upload=461218]: Waiting for error group managing segments... Oct 15 09:48:41.004514306 (524h17m26.466606789s) [noise=true , address=145.79.227.7:4294 , piece_id=H4ZNVIXORNO4MTXOVYDQZI4IJE4VXHZWC6JI35YKOR3UESSXJNTQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.006116557 (524h17m26.46820904s) [noise=true , address=145.79.227.5:4222 , piece_id=C2JSPN2QVSGSXG6ZW3ZMOFX5YFNN65XY7E62N5OEHOOMK3NSO4KQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.012963184 (524h17m26.475055667s) [noise=true , address=145.79.227.16:4910 , piece_id=BIHEJDH7RPI4UCPUNV453QHSEVCLA54PSNNQB2II37XZDRXFZNWQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.012994684 (524h17m26.475087167s) [noise=true , address=145.79.227.14:4840 , piece_id=B4ZVPRMVH7Y3TCWCLPO7DWRVMJL7VSI73CIT4PEHCMTIEURQKB2Q, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.016649956 (524h17m26.478742439s) [noise=true , address=145.79.227.11:4730 , piece_id=VO2DSM5LT43QAFPWOY3W3PVEJ44GZJWTCWG5QRQQFB363DFVFZDA, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.019248864 (524h17m26.481341347s) [noise=true , address=145.79.227.17:4969 , piece_id=R5HIP33QW5HLICARFWDUOW7I75YTDHZTJCIML5X52VKHLPL3J4YQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.075991585 (524h17m26.538084068s) [noise=true , address=145.79.227.4:4187 , piece_id=TQVPL6ZA6LHKAJVL6XWSYASZD3MCKKV3553CRFD72XWTVWY62STQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.118655146 (524h17m26.580747619s) [noise=true , address=145.79.227.14:4838 , piece_id=6K6A4CNEHGB3X6DEBLF5KHLUT4GE6BQBW6IE3RHBML2NWPN255AA, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.285934795 (524h17m26.748027278s) [noise=true , address=77.90.31.11:4555 , piece_id=U5RYZSD643KSUI4XLL26EAN6FQRPMG62GQMMCQRPEEG2XKSNBIZQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.744525465 (524h17m27.206617948s) [noise=true , address=45.91.116.9:4262 , piece_id=X53JZGRLZLX53UZLLCXXUQCFSQD2FO76X74OM45G26HNDU55UXXQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:42.218961989 (524h17m27.681054462s) [noise=true , address=45.91.116.11:4117 , piece_id=VH7ZJKSZ2NUXQJZE2IKNWP5PAL3TN2NNHD3ATZUQHGKKYMSSM3LQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:42.313638675 (524h17m27.775731158s) [noise=true , address=45.91.116.14:4189 , piece_id=UMQRQYPZ7SWUCO2N7JDCEVI5U5P36FDL3SUM6HIYJVWVOZTWNXQQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:42.434283559 (524h17m27.896376053s) [noise=true , address=45.91.116.8:4045 , piece_id=EDXBNLKKRZMT6MKK2KOIZM7DDKA3IHB6UMCXX4YKZF7JY2T3S5CQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:42.434321832 (524h17m27.896414305s) [seg_pos={0 0}, upload=461218]: Segment reached optimal threshold of 13 pieces. Oct 15 09:48:42.434605041 (524h17m27.896697534s) [noise=true , address=45.91.116.5:4299 , piece_id=4ARMXXE5LBRE2OZQEIITELKPTW4TISZWTD6XZGI4UJV7UCLUXAEQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX): protocol: expected piece hash; context canceled; EOF Oct 15 09:48:42.434666138 (524h17m27.896758621s) [noise=true , address=45.91.116.5:4299 , piece_id=4ARMXXE5LBRE2OZQEIITELKPTW4TISZWTD6XZGI4UJV7UCLUXAEQ, seg_pos={0 0}, upload=461218]: Long tail cancellation due to optimal threshold Oct 15 09:48:42.434692849 (524h17m27.896785432s) [noise=true , address=45.91.116.12:4226 , piece_id=DRXRGO7OXEKVS5ACL6FC7RKLGBBD6KNI7X6E5DM5D5VV5PEKN7FQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1): protocol: expected piece hash; context canceled; EOF Oct 15 09:48:42.434706184 (524h17m27.896798657s) [noise=true , address=45.91.116.12:4226 , piece_id=DRXRGO7OXEKVS5ACL6FC7RKLGBBD6KNI7X6E5DM5D5VV5PEKN7FQ, seg_pos={0 0}, upload=461218]: Long tail cancellation due to optimal threshold Oct 15 09:48:42.434718318 (524h17m27.896810911s) [noise=true , address=45.91.116.4:4940 , piece_id=3MCSUDYN73NIVJTGKPOVY6N3MALB3GBXI7I5NAX4HXPDHVMQ52BA, seg_pos={0 0}, upload=461218]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe): protocol: expected piece hash; context canceled; EOF Oct 15 09:48:42.434728447 (524h17m27.89682093s) [noise=true , address=45.91.116.4:4940 , piece_id=3MCSUDYN73NIVJTGKPOVY6N3MALB3GBXI7I5NAX4HXPDHVMQ52BA, seg_pos={0 0}, upload=461218]: Long tail cancellation due to optimal threshold Oct 15 09:48:42.434760458 (524h17m27.896852931s) [seg_pos={0 0}, upload=461218]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 15 09:48:42.434866900 (524h17m27.896959384s) [upload=461218]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 15 09:48:42.434873884 (524h17m27.896966367s) [upload=461218]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 15 09:48:42.760633614 (524h17m28.222726087s) [upload=461218]: Done uploading segments... Oct 15 09:48:42.760655435 (524h17m28.222747918s) [upload=461218]: Upload finished. err: <nil> Duration: 6.329407987s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.........................X...X..............................] duration: 5.904105299s FAIL (2025-10-15 10:09:41 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, Piece ID: VH7ZJKSZ2NUXQJZE2IKNWP5PAL3TN2NNHD3ATZUQHGKKYMSSM3LQ) context deadline exceeded; piecestore: (Node ID: 1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N, Piece ID: UMQRQYPZ7SWUCO2N7JDCEVI5U5P36FDL3SUM6HIYJVWVOZTWNXQQ) context deadline exceeded ======LOGS====== Oct 15 09:48:38.801058027 (524h17m24.26315051s) [upload=461218]: Starting upload Oct 15 09:48:38.801138119 (524h17m24.263230592s) [upload=461218]: Done starting upload Oct 15 09:48:38.801288545 (524h17m24.263381028s) [upload=461218]: Uploading segments... Oct 15 09:48:38.801442509 (524h17m24.263535002s) [upload=461218]: Got next segment. Inline: false Oct 15 09:48:38.801486663 (524h17m24.263579136s) [upload=461218]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 15 09:48:38.801493245 (524h17m24.263585728s) [upload=461218]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 15 09:48:39.066131918 (524h17m24.528224401s) [seg_pos={0 0}, upload=461218]: Begin upload segment... Oct 15 09:48:39.066205549 (524h17m24.528298022s) [seg_pos={0 0}, upload=461218]: Done begin upload segment. Oct 15 09:48:39.066243090 (524h17m24.528335573s) [noise=true , address=45.91.116.5:4299 , piece_id=4ARMXXE5LBRE2OZQEIITELKPTW4TISZWTD6XZGI4UJV7UCLUXAEQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.066254431 (524h17m24.528346915s) [noise=true , address=45.91.116.11:4117 , piece_id=VH7ZJKSZ2NUXQJZE2IKNWP5PAL3TN2NNHD3ATZUQHGKKYMSSM3LQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.066874873 (524h17m24.528967356s) [noise=true , address=145.79.227.7:4294 , piece_id=H4ZNVIXORNO4MTXOVYDQZI4IJE4VXHZWC6JI35YKOR3UESSXJNTQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.067301676 (524h17m24.529394159s) [noise=true , address=45.91.116.9:4262 , piece_id=X53JZGRLZLX53UZLLCXXUQCFSQD2FO76X74OM45G26HNDU55UXXQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.067705735 (524h17m24.529798218s) [noise=true , address=145.79.227.4:4187 , piece_id=TQVPL6ZA6LHKAJVL6XWSYASZD3MCKKV3553CRFD72XWTVWY62STQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.069183369 (524h17m24.531275862s) [noise=true , address=145.79.227.5:4222 , piece_id=C2JSPN2QVSGSXG6ZW3ZMOFX5YFNN65XY7E62N5OEHOOMK3NSO4KQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.070534782 (524h17m24.532627265s) [noise=true , address=77.90.31.11:4555 , piece_id=U5RYZSD643KSUI4XLL26EAN6FQRPMG62GQMMCQRPEEG2XKSNBIZQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.072356982 (524h17m24.534449466s) [noise=true , address=145.79.227.11:4730 , piece_id=VO2DSM5LT43QAFPWOY3W3PVEJ44GZJWTCWG5QRQQFB363DFVFZDA, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.073961167 (524h17m24.53605364s) [noise=true , address=45.91.116.4:4940 , piece_id=3MCSUDYN73NIVJTGKPOVY6N3MALB3GBXI7I5NAX4HXPDHVMQ52BA, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.075716960 (524h17m24.537809433s) [noise=true , address=145.79.227.16:4910 , piece_id=BIHEJDH7RPI4UCPUNV453QHSEVCLA54PSNNQB2II37XZDRXFZNWQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.079266419 (524h17m24.541358902s) [noise=true , address=145.79.227.14:4838 , piece_id=6K6A4CNEHGB3X6DEBLF5KHLUT4GE6BQBW6IE3RHBML2NWPN255AA, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.080943032 (524h17m24.543035505s) [noise=true , address=45.91.116.12:4226 , piece_id=DRXRGO7OXEKVS5ACL6FC7RKLGBBD6KNI7X6E5DM5D5VV5PEKN7FQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.081340479 (524h17m24.543432962s) [noise=true , address=45.91.116.8:4045 , piece_id=EDXBNLKKRZMT6MKK2KOIZM7DDKA3IHB6UMCXX4YKZF7JY2T3S5CQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.083616223 (524h17m24.545708706s) [noise=true , address=145.79.227.14:4840 , piece_id=B4ZVPRMVH7Y3TCWCLPO7DWRVMJL7VSI73CIT4PEHCMTIEURQKB2Q, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.083633946 (524h17m24.545726429s) [noise=true , address=45.91.116.14:4189 , piece_id=UMQRQYPZ7SWUCO2N7JDCEVI5U5P36FDL3SUM6HIYJVWVOZTWNXQQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.085493247 (524h17m24.54758573s) [noise=true , address=145.79.227.17:4969 , piece_id=R5HIP33QW5HLICARFWDUOW7I75YTDHZTJCIML5X52VKHLPL3J4YQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:40.727263388 (524h17m26.189355871s) [upload=461218]: Next returned nil segment Oct 15 09:48:40.727310076 (524h17m26.189402549s) [upload=461218]: Waiting for error group managing segments... Oct 15 09:48:41.004514306 (524h17m26.466606789s) [noise=true , address=145.79.227.7:4294 , piece_id=H4ZNVIXORNO4MTXOVYDQZI4IJE4VXHZWC6JI35YKOR3UESSXJNTQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.006116557 (524h17m26.46820904s) [noise=true , address=145.79.227.5:4222 , piece_id=C2JSPN2QVSGSXG6ZW3ZMOFX5YFNN65XY7E62N5OEHOOMK3NSO4KQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.012963184 (524h17m26.475055667s) [noise=true , address=145.79.227.16:4910 , piece_id=BIHEJDH7RPI4UCPUNV453QHSEVCLA54PSNNQB2II37XZDRXFZNWQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.012994684 (524h17m26.475087167s) [noise=true , address=145.79.227.14:4840 , piece_id=B4ZVPRMVH7Y3TCWCLPO7DWRVMJL7VSI73CIT4PEHCMTIEURQKB2Q, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.016649956 (524h17m26.478742439s) [noise=true , address=145.79.227.11:4730 , piece_id=VO2DSM5LT43QAFPWOY3W3PVEJ44GZJWTCWG5QRQQFB363DFVFZDA, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.019248864 (524h17m26.481341347s) [noise=true , address=145.79.227.17:4969 , piece_id=R5HIP33QW5HLICARFWDUOW7I75YTDHZTJCIML5X52VKHLPL3J4YQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.075991585 (524h17m26.538084068s) [noise=true , address=145.79.227.4:4187 , piece_id=TQVPL6ZA6LHKAJVL6XWSYASZD3MCKKV3553CRFD72XWTVWY62STQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.118655146 (524h17m26.580747619s) [noise=true , address=145.79.227.14:4838 , piece_id=6K6A4CNEHGB3X6DEBLF5KHLUT4GE6BQBW6IE3RHBML2NWPN255AA, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.285934795 (524h17m26.748027278s) [noise=true , address=77.90.31.11:4555 , piece_id=U5RYZSD643KSUI4XLL26EAN6FQRPMG62GQMMCQRPEEG2XKSNBIZQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.744525465 (524h17m27.206617948s) [noise=true , address=45.91.116.9:4262 , piece_id=X53JZGRLZLX53UZLLCXXUQCFSQD2FO76X74OM45G26HNDU55UXXQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:42.218961989 (524h17m27.681054462s) [noise=true , address=45.91.116.11:4117 , piece_id=VH7ZJKSZ2NUXQJZE2IKNWP5PAL3TN2NNHD3ATZUQHGKKYMSSM3LQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:42.313638675 (524h17m27.775731158s) [noise=true , address=45.91.116.14:4189 , piece_id=UMQRQYPZ7SWUCO2N7JDCEVI5U5P36FDL3SUM6HIYJVWVOZTWNXQQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:42.434283559 (524h17m27.896376053s) [noise=true , address=45.91.116.8:4045 , piece_id=EDXBNLKKRZMT6MKK2KOIZM7DDKA3IHB6UMCXX4YKZF7JY2T3S5CQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:42.434321832 (524h17m27.896414305s) [seg_pos={0 0}, upload=461218]: Segment reached optimal threshold of 13 pieces. Oct 15 09:48:42.434605041 (524h17m27.896697534s) [noise=true , address=45.91.116.5:4299 , piece_id=4ARMXXE5LBRE2OZQEIITELKPTW4TISZWTD6XZGI4UJV7UCLUXAEQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX): protocol: expected piece hash; context canceled; EOF Oct 15 09:48:42.434666138 (524h17m27.896758621s) [noise=true , address=45.91.116.5:4299 , piece_id=4ARMXXE5LBRE2OZQEIITELKPTW4TISZWTD6XZGI4UJV7UCLUXAEQ, seg_pos={0 0}, upload=461218]: Long tail cancellation due to optimal threshold Oct 15 09:48:42.434692849 (524h17m27.896785432s) [noise=true , address=45.91.116.12:4226 , piece_id=DRXRGO7OXEKVS5ACL6FC7RKLGBBD6KNI7X6E5DM5D5VV5PEKN7FQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1): protocol: expected piece hash; context canceled; EOF Oct 15 09:48:42.434706184 (524h17m27.896798657s) [noise=true , address=45.91.116.12:4226 , piece_id=DRXRGO7OXEKVS5ACL6FC7RKLGBBD6KNI7X6E5DM5D5VV5PEKN7FQ, seg_pos={0 0}, upload=461218]: Long tail cancellation due to optimal threshold Oct 15 09:48:42.434718318 (524h17m27.896810911s) [noise=true , address=45.91.116.4:4940 , piece_id=3MCSUDYN73NIVJTGKPOVY6N3MALB3GBXI7I5NAX4HXPDHVMQ52BA, seg_pos={0 0}, upload=461218]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe): protocol: expected piece hash; context canceled; EOF Oct 15 09:48:42.434728447 (524h17m27.89682093s) [noise=true , address=45.91.116.4:4940 , piece_id=3MCSUDYN73NIVJTGKPOVY6N3MALB3GBXI7I5NAX4HXPDHVMQ52BA, seg_pos={0 0}, upload=461218]: Long tail cancellation due to optimal threshold Oct 15 09:48:42.434760458 (524h17m27.896852931s) [seg_pos={0 0}, upload=461218]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 15 09:48:42.434866900 (524h17m27.896959384s) [upload=461218]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 15 09:48:42.434873884 (524h17m27.896966367s) [upload=461218]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 15 09:48:42.760633614 (524h17m28.222726087s) [upload=461218]: Done uploading segments... Oct 15 09:48:42.760655435 (524h17m28.222747918s) [upload=461218]: Upload finished. err: <nil> Duration: 5.904139485s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..........................X...X.............................] duration: 6.052472084s FAIL (2025-10-15 10:09:10 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, Piece ID: VH7ZJKSZ2NUXQJZE2IKNWP5PAL3TN2NNHD3ATZUQHGKKYMSSM3LQ) context deadline exceeded; piecestore: (Node ID: 1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N, Piece ID: UMQRQYPZ7SWUCO2N7JDCEVI5U5P36FDL3SUM6HIYJVWVOZTWNXQQ) context deadline exceeded ======LOGS====== Oct 15 09:48:38.801058027 (524h17m24.26315051s) [upload=461218]: Starting upload Oct 15 09:48:38.801138119 (524h17m24.263230592s) [upload=461218]: Done starting upload Oct 15 09:48:38.801288545 (524h17m24.263381028s) [upload=461218]: Uploading segments... Oct 15 09:48:38.801442509 (524h17m24.263535002s) [upload=461218]: Got next segment. Inline: false Oct 15 09:48:38.801486663 (524h17m24.263579136s) [upload=461218]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 15 09:48:38.801493245 (524h17m24.263585728s) [upload=461218]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 15 09:48:39.066131918 (524h17m24.528224401s) [seg_pos={0 0}, upload=461218]: Begin upload segment... Oct 15 09:48:39.066205549 (524h17m24.528298022s) [seg_pos={0 0}, upload=461218]: Done begin upload segment. Oct 15 09:48:39.066243090 (524h17m24.528335573s) [noise=true , address=45.91.116.5:4299 , piece_id=4ARMXXE5LBRE2OZQEIITELKPTW4TISZWTD6XZGI4UJV7UCLUXAEQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.066254431 (524h17m24.528346915s) [noise=true , address=45.91.116.11:4117 , piece_id=VH7ZJKSZ2NUXQJZE2IKNWP5PAL3TN2NNHD3ATZUQHGKKYMSSM3LQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.066874873 (524h17m24.528967356s) [noise=true , address=145.79.227.7:4294 , piece_id=H4ZNVIXORNO4MTXOVYDQZI4IJE4VXHZWC6JI35YKOR3UESSXJNTQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.067301676 (524h17m24.529394159s) [noise=true , address=45.91.116.9:4262 , piece_id=X53JZGRLZLX53UZLLCXXUQCFSQD2FO76X74OM45G26HNDU55UXXQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.067705735 (524h17m24.529798218s) [noise=true , address=145.79.227.4:4187 , piece_id=TQVPL6ZA6LHKAJVL6XWSYASZD3MCKKV3553CRFD72XWTVWY62STQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.069183369 (524h17m24.531275862s) [noise=true , address=145.79.227.5:4222 , piece_id=C2JSPN2QVSGSXG6ZW3ZMOFX5YFNN65XY7E62N5OEHOOMK3NSO4KQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.070534782 (524h17m24.532627265s) [noise=true , address=77.90.31.11:4555 , piece_id=U5RYZSD643KSUI4XLL26EAN6FQRPMG62GQMMCQRPEEG2XKSNBIZQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.072356982 (524h17m24.534449466s) [noise=true , address=145.79.227.11:4730 , piece_id=VO2DSM5LT43QAFPWOY3W3PVEJ44GZJWTCWG5QRQQFB363DFVFZDA, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.073961167 (524h17m24.53605364s) [noise=true , address=45.91.116.4:4940 , piece_id=3MCSUDYN73NIVJTGKPOVY6N3MALB3GBXI7I5NAX4HXPDHVMQ52BA, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.075716960 (524h17m24.537809433s) [noise=true , address=145.79.227.16:4910 , piece_id=BIHEJDH7RPI4UCPUNV453QHSEVCLA54PSNNQB2II37XZDRXFZNWQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.079266419 (524h17m24.541358902s) [noise=true , address=145.79.227.14:4838 , piece_id=6K6A4CNEHGB3X6DEBLF5KHLUT4GE6BQBW6IE3RHBML2NWPN255AA, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.080943032 (524h17m24.543035505s) [noise=true , address=45.91.116.12:4226 , piece_id=DRXRGO7OXEKVS5ACL6FC7RKLGBBD6KNI7X6E5DM5D5VV5PEKN7FQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.081340479 (524h17m24.543432962s) [noise=true , address=45.91.116.8:4045 , piece_id=EDXBNLKKRZMT6MKK2KOIZM7DDKA3IHB6UMCXX4YKZF7JY2T3S5CQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.083616223 (524h17m24.545708706s) [noise=true , address=145.79.227.14:4840 , piece_id=B4ZVPRMVH7Y3TCWCLPO7DWRVMJL7VSI73CIT4PEHCMTIEURQKB2Q, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.083633946 (524h17m24.545726429s) [noise=true , address=45.91.116.14:4189 , piece_id=UMQRQYPZ7SWUCO2N7JDCEVI5U5P36FDL3SUM6HIYJVWVOZTWNXQQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.085493247 (524h17m24.54758573s) [noise=true , address=145.79.227.17:4969 , piece_id=R5HIP33QW5HLICARFWDUOW7I75YTDHZTJCIML5X52VKHLPL3J4YQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:40.727263388 (524h17m26.189355871s) [upload=461218]: Next returned nil segment Oct 15 09:48:40.727310076 (524h17m26.189402549s) [upload=461218]: Waiting for error group managing segments... Oct 15 09:48:41.004514306 (524h17m26.466606789s) [noise=true , address=145.79.227.7:4294 , piece_id=H4ZNVIXORNO4MTXOVYDQZI4IJE4VXHZWC6JI35YKOR3UESSXJNTQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.006116557 (524h17m26.46820904s) [noise=true , address=145.79.227.5:4222 , piece_id=C2JSPN2QVSGSXG6ZW3ZMOFX5YFNN65XY7E62N5OEHOOMK3NSO4KQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.012963184 (524h17m26.475055667s) [noise=true , address=145.79.227.16:4910 , piece_id=BIHEJDH7RPI4UCPUNV453QHSEVCLA54PSNNQB2II37XZDRXFZNWQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.012994684 (524h17m26.475087167s) [noise=true , address=145.79.227.14:4840 , piece_id=B4ZVPRMVH7Y3TCWCLPO7DWRVMJL7VSI73CIT4PEHCMTIEURQKB2Q, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.016649956 (524h17m26.478742439s) [noise=true , address=145.79.227.11:4730 , piece_id=VO2DSM5LT43QAFPWOY3W3PVEJ44GZJWTCWG5QRQQFB363DFVFZDA, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.019248864 (524h17m26.481341347s) [noise=true , address=145.79.227.17:4969 , piece_id=R5HIP33QW5HLICARFWDUOW7I75YTDHZTJCIML5X52VKHLPL3J4YQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.075991585 (524h17m26.538084068s) [noise=true , address=145.79.227.4:4187 , piece_id=TQVPL6ZA6LHKAJVL6XWSYASZD3MCKKV3553CRFD72XWTVWY62STQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.118655146 (524h17m26.580747619s) [noise=true , address=145.79.227.14:4838 , piece_id=6K6A4CNEHGB3X6DEBLF5KHLUT4GE6BQBW6IE3RHBML2NWPN255AA, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.285934795 (524h17m26.748027278s) [noise=true , address=77.90.31.11:4555 , piece_id=U5RYZSD643KSUI4XLL26EAN6FQRPMG62GQMMCQRPEEG2XKSNBIZQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.744525465 (524h17m27.206617948s) [noise=true , address=45.91.116.9:4262 , piece_id=X53JZGRLZLX53UZLLCXXUQCFSQD2FO76X74OM45G26HNDU55UXXQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:42.218961989 (524h17m27.681054462s) [noise=true , address=45.91.116.11:4117 , piece_id=VH7ZJKSZ2NUXQJZE2IKNWP5PAL3TN2NNHD3ATZUQHGKKYMSSM3LQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:42.313638675 (524h17m27.775731158s) [noise=true , address=45.91.116.14:4189 , piece_id=UMQRQYPZ7SWUCO2N7JDCEVI5U5P36FDL3SUM6HIYJVWVOZTWNXQQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:42.434283559 (524h17m27.896376053s) [noise=true , address=45.91.116.8:4045 , piece_id=EDXBNLKKRZMT6MKK2KOIZM7DDKA3IHB6UMCXX4YKZF7JY2T3S5CQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:42.434321832 (524h17m27.896414305s) [seg_pos={0 0}, upload=461218]: Segment reached optimal threshold of 13 pieces. Oct 15 09:48:42.434605041 (524h17m27.896697534s) [noise=true , address=45.91.116.5:4299 , piece_id=4ARMXXE5LBRE2OZQEIITELKPTW4TISZWTD6XZGI4UJV7UCLUXAEQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX): protocol: expected piece hash; context canceled; EOF Oct 15 09:48:42.434666138 (524h17m27.896758621s) [noise=true , address=45.91.116.5:4299 , piece_id=4ARMXXE5LBRE2OZQEIITELKPTW4TISZWTD6XZGI4UJV7UCLUXAEQ, seg_pos={0 0}, upload=461218]: Long tail cancellation due to optimal threshold Oct 15 09:48:42.434692849 (524h17m27.896785432s) [noise=true , address=45.91.116.12:4226 , piece_id=DRXRGO7OXEKVS5ACL6FC7RKLGBBD6KNI7X6E5DM5D5VV5PEKN7FQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1): protocol: expected piece hash; context canceled; EOF Oct 15 09:48:42.434706184 (524h17m27.896798657s) [noise=true , address=45.91.116.12:4226 , piece_id=DRXRGO7OXEKVS5ACL6FC7RKLGBBD6KNI7X6E5DM5D5VV5PEKN7FQ, seg_pos={0 0}, upload=461218]: Long tail cancellation due to optimal threshold Oct 15 09:48:42.434718318 (524h17m27.896810911s) [noise=true , address=45.91.116.4:4940 , piece_id=3MCSUDYN73NIVJTGKPOVY6N3MALB3GBXI7I5NAX4HXPDHVMQ52BA, seg_pos={0 0}, upload=461218]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe): protocol: expected piece hash; context canceled; EOF Oct 15 09:48:42.434728447 (524h17m27.89682093s) [noise=true , address=45.91.116.4:4940 , piece_id=3MCSUDYN73NIVJTGKPOVY6N3MALB3GBXI7I5NAX4HXPDHVMQ52BA, seg_pos={0 0}, upload=461218]: Long tail cancellation due to optimal threshold Oct 15 09:48:42.434760458 (524h17m27.896852931s) [seg_pos={0 0}, upload=461218]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 15 09:48:42.434866900 (524h17m27.896959384s) [upload=461218]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 15 09:48:42.434873884 (524h17m27.896966367s) [upload=461218]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 15 09:48:42.760633614 (524h17m28.222726087s) [upload=461218]: Done uploading segments... Oct 15 09:48:42.760655435 (524h17m28.222747918s) [upload=461218]: Upload finished. err: <nil> Duration: 6.05257003s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...........................X...X............................] duration: 6.393403525s FAIL (2025-10-15 10:08:24 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, Piece ID: VH7ZJKSZ2NUXQJZE2IKNWP5PAL3TN2NNHD3ATZUQHGKKYMSSM3LQ) context deadline exceeded; piecestore: (Node ID: 1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N, Piece ID: UMQRQYPZ7SWUCO2N7JDCEVI5U5P36FDL3SUM6HIYJVWVOZTWNXQQ) context deadline exceeded ======LOGS====== Oct 15 09:48:38.801058027 (524h17m24.26315051s) [upload=461218]: Starting upload Oct 15 09:48:38.801138119 (524h17m24.263230592s) [upload=461218]: Done starting upload Oct 15 09:48:38.801288545 (524h17m24.263381028s) [upload=461218]: Uploading segments... Oct 15 09:48:38.801442509 (524h17m24.263535002s) [upload=461218]: Got next segment. Inline: false Oct 15 09:48:38.801486663 (524h17m24.263579136s) [upload=461218]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 15 09:48:38.801493245 (524h17m24.263585728s) [upload=461218]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 15 09:48:39.066131918 (524h17m24.528224401s) [seg_pos={0 0}, upload=461218]: Begin upload segment... Oct 15 09:48:39.066205549 (524h17m24.528298022s) [seg_pos={0 0}, upload=461218]: Done begin upload segment. Oct 15 09:48:39.066243090 (524h17m24.528335573s) [noise=true , address=45.91.116.5:4299 , piece_id=4ARMXXE5LBRE2OZQEIITELKPTW4TISZWTD6XZGI4UJV7UCLUXAEQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.066254431 (524h17m24.528346915s) [noise=true , address=45.91.116.11:4117 , piece_id=VH7ZJKSZ2NUXQJZE2IKNWP5PAL3TN2NNHD3ATZUQHGKKYMSSM3LQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.066874873 (524h17m24.528967356s) [noise=true , address=145.79.227.7:4294 , piece_id=H4ZNVIXORNO4MTXOVYDQZI4IJE4VXHZWC6JI35YKOR3UESSXJNTQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.067301676 (524h17m24.529394159s) [noise=true , address=45.91.116.9:4262 , piece_id=X53JZGRLZLX53UZLLCXXUQCFSQD2FO76X74OM45G26HNDU55UXXQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.067705735 (524h17m24.529798218s) [noise=true , address=145.79.227.4:4187 , piece_id=TQVPL6ZA6LHKAJVL6XWSYASZD3MCKKV3553CRFD72XWTVWY62STQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.069183369 (524h17m24.531275862s) [noise=true , address=145.79.227.5:4222 , piece_id=C2JSPN2QVSGSXG6ZW3ZMOFX5YFNN65XY7E62N5OEHOOMK3NSO4KQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.070534782 (524h17m24.532627265s) [noise=true , address=77.90.31.11:4555 , piece_id=U5RYZSD643KSUI4XLL26EAN6FQRPMG62GQMMCQRPEEG2XKSNBIZQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.072356982 (524h17m24.534449466s) [noise=true , address=145.79.227.11:4730 , piece_id=VO2DSM5LT43QAFPWOY3W3PVEJ44GZJWTCWG5QRQQFB363DFVFZDA, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.073961167 (524h17m24.53605364s) [noise=true , address=45.91.116.4:4940 , piece_id=3MCSUDYN73NIVJTGKPOVY6N3MALB3GBXI7I5NAX4HXPDHVMQ52BA, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.075716960 (524h17m24.537809433s) [noise=true , address=145.79.227.16:4910 , piece_id=BIHEJDH7RPI4UCPUNV453QHSEVCLA54PSNNQB2II37XZDRXFZNWQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.079266419 (524h17m24.541358902s) [noise=true , address=145.79.227.14:4838 , piece_id=6K6A4CNEHGB3X6DEBLF5KHLUT4GE6BQBW6IE3RHBML2NWPN255AA, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.080943032 (524h17m24.543035505s) [noise=true , address=45.91.116.12:4226 , piece_id=DRXRGO7OXEKVS5ACL6FC7RKLGBBD6KNI7X6E5DM5D5VV5PEKN7FQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.081340479 (524h17m24.543432962s) [noise=true , address=45.91.116.8:4045 , piece_id=EDXBNLKKRZMT6MKK2KOIZM7DDKA3IHB6UMCXX4YKZF7JY2T3S5CQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.083616223 (524h17m24.545708706s) [noise=true , address=145.79.227.14:4840 , piece_id=B4ZVPRMVH7Y3TCWCLPO7DWRVMJL7VSI73CIT4PEHCMTIEURQKB2Q, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.083633946 (524h17m24.545726429s) [noise=true , address=45.91.116.14:4189 , piece_id=UMQRQYPZ7SWUCO2N7JDCEVI5U5P36FDL3SUM6HIYJVWVOZTWNXQQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.085493247 (524h17m24.54758573s) [noise=true , address=145.79.227.17:4969 , piece_id=R5HIP33QW5HLICARFWDUOW7I75YTDHZTJCIML5X52VKHLPL3J4YQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:40.727263388 (524h17m26.189355871s) [upload=461218]: Next returned nil segment Oct 15 09:48:40.727310076 (524h17m26.189402549s) [upload=461218]: Waiting for error group managing segments... Oct 15 09:48:41.004514306 (524h17m26.466606789s) [noise=true , address=145.79.227.7:4294 , piece_id=H4ZNVIXORNO4MTXOVYDQZI4IJE4VXHZWC6JI35YKOR3UESSXJNTQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.006116557 (524h17m26.46820904s) [noise=true , address=145.79.227.5:4222 , piece_id=C2JSPN2QVSGSXG6ZW3ZMOFX5YFNN65XY7E62N5OEHOOMK3NSO4KQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.012963184 (524h17m26.475055667s) [noise=true , address=145.79.227.16:4910 , piece_id=BIHEJDH7RPI4UCPUNV453QHSEVCLA54PSNNQB2II37XZDRXFZNWQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.012994684 (524h17m26.475087167s) [noise=true , address=145.79.227.14:4840 , piece_id=B4ZVPRMVH7Y3TCWCLPO7DWRVMJL7VSI73CIT4PEHCMTIEURQKB2Q, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.016649956 (524h17m26.478742439s) [noise=true , address=145.79.227.11:4730 , piece_id=VO2DSM5LT43QAFPWOY3W3PVEJ44GZJWTCWG5QRQQFB363DFVFZDA, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.019248864 (524h17m26.481341347s) [noise=true , address=145.79.227.17:4969 , piece_id=R5HIP33QW5HLICARFWDUOW7I75YTDHZTJCIML5X52VKHLPL3J4YQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.075991585 (524h17m26.538084068s) [noise=true , address=145.79.227.4:4187 , piece_id=TQVPL6ZA6LHKAJVL6XWSYASZD3MCKKV3553CRFD72XWTVWY62STQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.118655146 (524h17m26.580747619s) [noise=true , address=145.79.227.14:4838 , piece_id=6K6A4CNEHGB3X6DEBLF5KHLUT4GE6BQBW6IE3RHBML2NWPN255AA, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.285934795 (524h17m26.748027278s) [noise=true , address=77.90.31.11:4555 , piece_id=U5RYZSD643KSUI4XLL26EAN6FQRPMG62GQMMCQRPEEG2XKSNBIZQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.744525465 (524h17m27.206617948s) [noise=true , address=45.91.116.9:4262 , piece_id=X53JZGRLZLX53UZLLCXXUQCFSQD2FO76X74OM45G26HNDU55UXXQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:42.218961989 (524h17m27.681054462s) [noise=true , address=45.91.116.11:4117 , piece_id=VH7ZJKSZ2NUXQJZE2IKNWP5PAL3TN2NNHD3ATZUQHGKKYMSSM3LQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:42.313638675 (524h17m27.775731158s) [noise=true , address=45.91.116.14:4189 , piece_id=UMQRQYPZ7SWUCO2N7JDCEVI5U5P36FDL3SUM6HIYJVWVOZTWNXQQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:42.434283559 (524h17m27.896376053s) [noise=true , address=45.91.116.8:4045 , piece_id=EDXBNLKKRZMT6MKK2KOIZM7DDKA3IHB6UMCXX4YKZF7JY2T3S5CQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:42.434321832 (524h17m27.896414305s) [seg_pos={0 0}, upload=461218]: Segment reached optimal threshold of 13 pieces. Oct 15 09:48:42.434605041 (524h17m27.896697534s) [noise=true , address=45.91.116.5:4299 , piece_id=4ARMXXE5LBRE2OZQEIITELKPTW4TISZWTD6XZGI4UJV7UCLUXAEQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX): protocol: expected piece hash; context canceled; EOF Oct 15 09:48:42.434666138 (524h17m27.896758621s) [noise=true , address=45.91.116.5:4299 , piece_id=4ARMXXE5LBRE2OZQEIITELKPTW4TISZWTD6XZGI4UJV7UCLUXAEQ, seg_pos={0 0}, upload=461218]: Long tail cancellation due to optimal threshold Oct 15 09:48:42.434692849 (524h17m27.896785432s) [noise=true , address=45.91.116.12:4226 , piece_id=DRXRGO7OXEKVS5ACL6FC7RKLGBBD6KNI7X6E5DM5D5VV5PEKN7FQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1): protocol: expected piece hash; context canceled; EOF Oct 15 09:48:42.434706184 (524h17m27.896798657s) [noise=true , address=45.91.116.12:4226 , piece_id=DRXRGO7OXEKVS5ACL6FC7RKLGBBD6KNI7X6E5DM5D5VV5PEKN7FQ, seg_pos={0 0}, upload=461218]: Long tail cancellation due to optimal threshold Oct 15 09:48:42.434718318 (524h17m27.896810911s) [noise=true , address=45.91.116.4:4940 , piece_id=3MCSUDYN73NIVJTGKPOVY6N3MALB3GBXI7I5NAX4HXPDHVMQ52BA, seg_pos={0 0}, upload=461218]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe): protocol: expected piece hash; context canceled; EOF Oct 15 09:48:42.434728447 (524h17m27.89682093s) [noise=true , address=45.91.116.4:4940 , piece_id=3MCSUDYN73NIVJTGKPOVY6N3MALB3GBXI7I5NAX4HXPDHVMQ52BA, seg_pos={0 0}, upload=461218]: Long tail cancellation due to optimal threshold Oct 15 09:48:42.434760458 (524h17m27.896852931s) [seg_pos={0 0}, upload=461218]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 15 09:48:42.434866900 (524h17m27.896959384s) [upload=461218]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 15 09:48:42.434873884 (524h17m27.896966367s) [upload=461218]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 15 09:48:42.760633614 (524h17m28.222726087s) [upload=461218]: Done uploading segments... Oct 15 09:48:42.760655435 (524h17m28.222747918s) [upload=461218]: Upload finished. err: <nil> Duration: 6.393434724s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [............................X...X...........................] duration: 8.944282623s FAIL (2025-10-15 10:07:23 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, Piece ID: VH7ZJKSZ2NUXQJZE2IKNWP5PAL3TN2NNHD3ATZUQHGKKYMSSM3LQ) context deadline exceeded; piecestore: (Node ID: 1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N, Piece ID: UMQRQYPZ7SWUCO2N7JDCEVI5U5P36FDL3SUM6HIYJVWVOZTWNXQQ) context deadline exceeded ======LOGS====== Oct 15 09:48:38.801058027 (524h17m24.26315051s) [upload=461218]: Starting upload Oct 15 09:48:38.801138119 (524h17m24.263230592s) [upload=461218]: Done starting upload Oct 15 09:48:38.801288545 (524h17m24.263381028s) [upload=461218]: Uploading segments... Oct 15 09:48:38.801442509 (524h17m24.263535002s) [upload=461218]: Got next segment. Inline: false Oct 15 09:48:38.801486663 (524h17m24.263579136s) [upload=461218]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 15 09:48:38.801493245 (524h17m24.263585728s) [upload=461218]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 15 09:48:39.066131918 (524h17m24.528224401s) [seg_pos={0 0}, upload=461218]: Begin upload segment... Oct 15 09:48:39.066205549 (524h17m24.528298022s) [seg_pos={0 0}, upload=461218]: Done begin upload segment. Oct 15 09:48:39.066243090 (524h17m24.528335573s) [noise=true , address=45.91.116.5:4299 , piece_id=4ARMXXE5LBRE2OZQEIITELKPTW4TISZWTD6XZGI4UJV7UCLUXAEQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.066254431 (524h17m24.528346915s) [noise=true , address=45.91.116.11:4117 , piece_id=VH7ZJKSZ2NUXQJZE2IKNWP5PAL3TN2NNHD3ATZUQHGKKYMSSM3LQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.066874873 (524h17m24.528967356s) [noise=true , address=145.79.227.7:4294 , piece_id=H4ZNVIXORNO4MTXOVYDQZI4IJE4VXHZWC6JI35YKOR3UESSXJNTQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.067301676 (524h17m24.529394159s) [noise=true , address=45.91.116.9:4262 , piece_id=X53JZGRLZLX53UZLLCXXUQCFSQD2FO76X74OM45G26HNDU55UXXQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.067705735 (524h17m24.529798218s) [noise=true , address=145.79.227.4:4187 , piece_id=TQVPL6ZA6LHKAJVL6XWSYASZD3MCKKV3553CRFD72XWTVWY62STQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.069183369 (524h17m24.531275862s) [noise=true , address=145.79.227.5:4222 , piece_id=C2JSPN2QVSGSXG6ZW3ZMOFX5YFNN65XY7E62N5OEHOOMK3NSO4KQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.070534782 (524h17m24.532627265s) [noise=true , address=77.90.31.11:4555 , piece_id=U5RYZSD643KSUI4XLL26EAN6FQRPMG62GQMMCQRPEEG2XKSNBIZQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.072356982 (524h17m24.534449466s) [noise=true , address=145.79.227.11:4730 , piece_id=VO2DSM5LT43QAFPWOY3W3PVEJ44GZJWTCWG5QRQQFB363DFVFZDA, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.073961167 (524h17m24.53605364s) [noise=true , address=45.91.116.4:4940 , piece_id=3MCSUDYN73NIVJTGKPOVY6N3MALB3GBXI7I5NAX4HXPDHVMQ52BA, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.075716960 (524h17m24.537809433s) [noise=true , address=145.79.227.16:4910 , piece_id=BIHEJDH7RPI4UCPUNV453QHSEVCLA54PSNNQB2II37XZDRXFZNWQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.079266419 (524h17m24.541358902s) [noise=true , address=145.79.227.14:4838 , piece_id=6K6A4CNEHGB3X6DEBLF5KHLUT4GE6BQBW6IE3RHBML2NWPN255AA, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.080943032 (524h17m24.543035505s) [noise=true , address=45.91.116.12:4226 , piece_id=DRXRGO7OXEKVS5ACL6FC7RKLGBBD6KNI7X6E5DM5D5VV5PEKN7FQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.081340479 (524h17m24.543432962s) [noise=true , address=45.91.116.8:4045 , piece_id=EDXBNLKKRZMT6MKK2KOIZM7DDKA3IHB6UMCXX4YKZF7JY2T3S5CQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.083616223 (524h17m24.545708706s) [noise=true , address=145.79.227.14:4840 , piece_id=B4ZVPRMVH7Y3TCWCLPO7DWRVMJL7VSI73CIT4PEHCMTIEURQKB2Q, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.083633946 (524h17m24.545726429s) [noise=true , address=45.91.116.14:4189 , piece_id=UMQRQYPZ7SWUCO2N7JDCEVI5U5P36FDL3SUM6HIYJVWVOZTWNXQQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.085493247 (524h17m24.54758573s) [noise=true , address=145.79.227.17:4969 , piece_id=R5HIP33QW5HLICARFWDUOW7I75YTDHZTJCIML5X52VKHLPL3J4YQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:40.727263388 (524h17m26.189355871s) [upload=461218]: Next returned nil segment Oct 15 09:48:40.727310076 (524h17m26.189402549s) [upload=461218]: Waiting for error group managing segments... Oct 15 09:48:41.004514306 (524h17m26.466606789s) [noise=true , address=145.79.227.7:4294 , piece_id=H4ZNVIXORNO4MTXOVYDQZI4IJE4VXHZWC6JI35YKOR3UESSXJNTQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.006116557 (524h17m26.46820904s) [noise=true , address=145.79.227.5:4222 , piece_id=C2JSPN2QVSGSXG6ZW3ZMOFX5YFNN65XY7E62N5OEHOOMK3NSO4KQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.012963184 (524h17m26.475055667s) [noise=true , address=145.79.227.16:4910 , piece_id=BIHEJDH7RPI4UCPUNV453QHSEVCLA54PSNNQB2II37XZDRXFZNWQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.012994684 (524h17m26.475087167s) [noise=true , address=145.79.227.14:4840 , piece_id=B4ZVPRMVH7Y3TCWCLPO7DWRVMJL7VSI73CIT4PEHCMTIEURQKB2Q, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.016649956 (524h17m26.478742439s) [noise=true , address=145.79.227.11:4730 , piece_id=VO2DSM5LT43QAFPWOY3W3PVEJ44GZJWTCWG5QRQQFB363DFVFZDA, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.019248864 (524h17m26.481341347s) [noise=true , address=145.79.227.17:4969 , piece_id=R5HIP33QW5HLICARFWDUOW7I75YTDHZTJCIML5X52VKHLPL3J4YQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.075991585 (524h17m26.538084068s) [noise=true , address=145.79.227.4:4187 , piece_id=TQVPL6ZA6LHKAJVL6XWSYASZD3MCKKV3553CRFD72XWTVWY62STQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.118655146 (524h17m26.580747619s) [noise=true , address=145.79.227.14:4838 , piece_id=6K6A4CNEHGB3X6DEBLF5KHLUT4GE6BQBW6IE3RHBML2NWPN255AA, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.285934795 (524h17m26.748027278s) [noise=true , address=77.90.31.11:4555 , piece_id=U5RYZSD643KSUI4XLL26EAN6FQRPMG62GQMMCQRPEEG2XKSNBIZQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.744525465 (524h17m27.206617948s) [noise=true , address=45.91.116.9:4262 , piece_id=X53JZGRLZLX53UZLLCXXUQCFSQD2FO76X74OM45G26HNDU55UXXQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:42.218961989 (524h17m27.681054462s) [noise=true , address=45.91.116.11:4117 , piece_id=VH7ZJKSZ2NUXQJZE2IKNWP5PAL3TN2NNHD3ATZUQHGKKYMSSM3LQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:42.313638675 (524h17m27.775731158s) [noise=true , address=45.91.116.14:4189 , piece_id=UMQRQYPZ7SWUCO2N7JDCEVI5U5P36FDL3SUM6HIYJVWVOZTWNXQQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:42.434283559 (524h17m27.896376053s) [noise=true , address=45.91.116.8:4045 , piece_id=EDXBNLKKRZMT6MKK2KOIZM7DDKA3IHB6UMCXX4YKZF7JY2T3S5CQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:42.434321832 (524h17m27.896414305s) [seg_pos={0 0}, upload=461218]: Segment reached optimal threshold of 13 pieces. Oct 15 09:48:42.434605041 (524h17m27.896697534s) [noise=true , address=45.91.116.5:4299 , piece_id=4ARMXXE5LBRE2OZQEIITELKPTW4TISZWTD6XZGI4UJV7UCLUXAEQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX): protocol: expected piece hash; context canceled; EOF Oct 15 09:48:42.434666138 (524h17m27.896758621s) [noise=true , address=45.91.116.5:4299 , piece_id=4ARMXXE5LBRE2OZQEIITELKPTW4TISZWTD6XZGI4UJV7UCLUXAEQ, seg_pos={0 0}, upload=461218]: Long tail cancellation due to optimal threshold Oct 15 09:48:42.434692849 (524h17m27.896785432s) [noise=true , address=45.91.116.12:4226 , piece_id=DRXRGO7OXEKVS5ACL6FC7RKLGBBD6KNI7X6E5DM5D5VV5PEKN7FQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1): protocol: expected piece hash; context canceled; EOF Oct 15 09:48:42.434706184 (524h17m27.896798657s) [noise=true , address=45.91.116.12:4226 , piece_id=DRXRGO7OXEKVS5ACL6FC7RKLGBBD6KNI7X6E5DM5D5VV5PEKN7FQ, seg_pos={0 0}, upload=461218]: Long tail cancellation due to optimal threshold Oct 15 09:48:42.434718318 (524h17m27.896810911s) [noise=true , address=45.91.116.4:4940 , piece_id=3MCSUDYN73NIVJTGKPOVY6N3MALB3GBXI7I5NAX4HXPDHVMQ52BA, seg_pos={0 0}, upload=461218]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe): protocol: expected piece hash; context canceled; EOF Oct 15 09:48:42.434728447 (524h17m27.89682093s) [noise=true , address=45.91.116.4:4940 , piece_id=3MCSUDYN73NIVJTGKPOVY6N3MALB3GBXI7I5NAX4HXPDHVMQ52BA, seg_pos={0 0}, upload=461218]: Long tail cancellation due to optimal threshold Oct 15 09:48:42.434760458 (524h17m27.896852931s) [seg_pos={0 0}, upload=461218]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 15 09:48:42.434866900 (524h17m27.896959384s) [upload=461218]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 15 09:48:42.434873884 (524h17m27.896966367s) [upload=461218]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 15 09:48:42.760633614 (524h17m28.222726087s) [upload=461218]: Done uploading segments... Oct 15 09:48:42.760655435 (524h17m28.222747918s) [upload=461218]: Upload finished. err: <nil> Duration: 8.944309435s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.............................X...X..........................] duration: 6.711143234s FAIL (2025-10-15 10:06:08 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, Piece ID: VH7ZJKSZ2NUXQJZE2IKNWP5PAL3TN2NNHD3ATZUQHGKKYMSSM3LQ) context deadline exceeded; piecestore: (Node ID: 1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N, Piece ID: UMQRQYPZ7SWUCO2N7JDCEVI5U5P36FDL3SUM6HIYJVWVOZTWNXQQ) context deadline exceeded ======LOGS====== Oct 15 09:48:38.801058027 (524h17m24.26315051s) [upload=461218]: Starting upload Oct 15 09:48:38.801138119 (524h17m24.263230592s) [upload=461218]: Done starting upload Oct 15 09:48:38.801288545 (524h17m24.263381028s) [upload=461218]: Uploading segments... Oct 15 09:48:38.801442509 (524h17m24.263535002s) [upload=461218]: Got next segment. Inline: false Oct 15 09:48:38.801486663 (524h17m24.263579136s) [upload=461218]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 15 09:48:38.801493245 (524h17m24.263585728s) [upload=461218]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 15 09:48:39.066131918 (524h17m24.528224401s) [seg_pos={0 0}, upload=461218]: Begin upload segment... Oct 15 09:48:39.066205549 (524h17m24.528298022s) [seg_pos={0 0}, upload=461218]: Done begin upload segment. Oct 15 09:48:39.066243090 (524h17m24.528335573s) [noise=true , address=45.91.116.5:4299 , piece_id=4ARMXXE5LBRE2OZQEIITELKPTW4TISZWTD6XZGI4UJV7UCLUXAEQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.066254431 (524h17m24.528346915s) [noise=true , address=45.91.116.11:4117 , piece_id=VH7ZJKSZ2NUXQJZE2IKNWP5PAL3TN2NNHD3ATZUQHGKKYMSSM3LQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.066874873 (524h17m24.528967356s) [noise=true , address=145.79.227.7:4294 , piece_id=H4ZNVIXORNO4MTXOVYDQZI4IJE4VXHZWC6JI35YKOR3UESSXJNTQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.067301676 (524h17m24.529394159s) [noise=true , address=45.91.116.9:4262 , piece_id=X53JZGRLZLX53UZLLCXXUQCFSQD2FO76X74OM45G26HNDU55UXXQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.067705735 (524h17m24.529798218s) [noise=true , address=145.79.227.4:4187 , piece_id=TQVPL6ZA6LHKAJVL6XWSYASZD3MCKKV3553CRFD72XWTVWY62STQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.069183369 (524h17m24.531275862s) [noise=true , address=145.79.227.5:4222 , piece_id=C2JSPN2QVSGSXG6ZW3ZMOFX5YFNN65XY7E62N5OEHOOMK3NSO4KQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.070534782 (524h17m24.532627265s) [noise=true , address=77.90.31.11:4555 , piece_id=U5RYZSD643KSUI4XLL26EAN6FQRPMG62GQMMCQRPEEG2XKSNBIZQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.072356982 (524h17m24.534449466s) [noise=true , address=145.79.227.11:4730 , piece_id=VO2DSM5LT43QAFPWOY3W3PVEJ44GZJWTCWG5QRQQFB363DFVFZDA, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.073961167 (524h17m24.53605364s) [noise=true , address=45.91.116.4:4940 , piece_id=3MCSUDYN73NIVJTGKPOVY6N3MALB3GBXI7I5NAX4HXPDHVMQ52BA, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.075716960 (524h17m24.537809433s) [noise=true , address=145.79.227.16:4910 , piece_id=BIHEJDH7RPI4UCPUNV453QHSEVCLA54PSNNQB2II37XZDRXFZNWQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.079266419 (524h17m24.541358902s) [noise=true , address=145.79.227.14:4838 , piece_id=6K6A4CNEHGB3X6DEBLF5KHLUT4GE6BQBW6IE3RHBML2NWPN255AA, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.080943032 (524h17m24.543035505s) [noise=true , address=45.91.116.12:4226 , piece_id=DRXRGO7OXEKVS5ACL6FC7RKLGBBD6KNI7X6E5DM5D5VV5PEKN7FQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.081340479 (524h17m24.543432962s) [noise=true , address=45.91.116.8:4045 , piece_id=EDXBNLKKRZMT6MKK2KOIZM7DDKA3IHB6UMCXX4YKZF7JY2T3S5CQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.083616223 (524h17m24.545708706s) [noise=true , address=145.79.227.14:4840 , piece_id=B4ZVPRMVH7Y3TCWCLPO7DWRVMJL7VSI73CIT4PEHCMTIEURQKB2Q, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.083633946 (524h17m24.545726429s) [noise=true , address=45.91.116.14:4189 , piece_id=UMQRQYPZ7SWUCO2N7JDCEVI5U5P36FDL3SUM6HIYJVWVOZTWNXQQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.085493247 (524h17m24.54758573s) [noise=true , address=145.79.227.17:4969 , piece_id=R5HIP33QW5HLICARFWDUOW7I75YTDHZTJCIML5X52VKHLPL3J4YQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:40.727263388 (524h17m26.189355871s) [upload=461218]: Next returned nil segment Oct 15 09:48:40.727310076 (524h17m26.189402549s) [upload=461218]: Waiting for error group managing segments... Oct 15 09:48:41.004514306 (524h17m26.466606789s) [noise=true , address=145.79.227.7:4294 , piece_id=H4ZNVIXORNO4MTXOVYDQZI4IJE4VXHZWC6JI35YKOR3UESSXJNTQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.006116557 (524h17m26.46820904s) [noise=true , address=145.79.227.5:4222 , piece_id=C2JSPN2QVSGSXG6ZW3ZMOFX5YFNN65XY7E62N5OEHOOMK3NSO4KQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.012963184 (524h17m26.475055667s) [noise=true , address=145.79.227.16:4910 , piece_id=BIHEJDH7RPI4UCPUNV453QHSEVCLA54PSNNQB2II37XZDRXFZNWQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.012994684 (524h17m26.475087167s) [noise=true , address=145.79.227.14:4840 , piece_id=B4ZVPRMVH7Y3TCWCLPO7DWRVMJL7VSI73CIT4PEHCMTIEURQKB2Q, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.016649956 (524h17m26.478742439s) [noise=true , address=145.79.227.11:4730 , piece_id=VO2DSM5LT43QAFPWOY3W3PVEJ44GZJWTCWG5QRQQFB363DFVFZDA, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.019248864 (524h17m26.481341347s) [noise=true , address=145.79.227.17:4969 , piece_id=R5HIP33QW5HLICARFWDUOW7I75YTDHZTJCIML5X52VKHLPL3J4YQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.075991585 (524h17m26.538084068s) [noise=true , address=145.79.227.4:4187 , piece_id=TQVPL6ZA6LHKAJVL6XWSYASZD3MCKKV3553CRFD72XWTVWY62STQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.118655146 (524h17m26.580747619s) [noise=true , address=145.79.227.14:4838 , piece_id=6K6A4CNEHGB3X6DEBLF5KHLUT4GE6BQBW6IE3RHBML2NWPN255AA, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.285934795 (524h17m26.748027278s) [noise=true , address=77.90.31.11:4555 , piece_id=U5RYZSD643KSUI4XLL26EAN6FQRPMG62GQMMCQRPEEG2XKSNBIZQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.744525465 (524h17m27.206617948s) [noise=true , address=45.91.116.9:4262 , piece_id=X53JZGRLZLX53UZLLCXXUQCFSQD2FO76X74OM45G26HNDU55UXXQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:42.218961989 (524h17m27.681054462s) [noise=true , address=45.91.116.11:4117 , piece_id=VH7ZJKSZ2NUXQJZE2IKNWP5PAL3TN2NNHD3ATZUQHGKKYMSSM3LQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:42.313638675 (524h17m27.775731158s) [noise=true , address=45.91.116.14:4189 , piece_id=UMQRQYPZ7SWUCO2N7JDCEVI5U5P36FDL3SUM6HIYJVWVOZTWNXQQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:42.434283559 (524h17m27.896376053s) [noise=true , address=45.91.116.8:4045 , piece_id=EDXBNLKKRZMT6MKK2KOIZM7DDKA3IHB6UMCXX4YKZF7JY2T3S5CQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:42.434321832 (524h17m27.896414305s) [seg_pos={0 0}, upload=461218]: Segment reached optimal threshold of 13 pieces. Oct 15 09:48:42.434605041 (524h17m27.896697534s) [noise=true , address=45.91.116.5:4299 , piece_id=4ARMXXE5LBRE2OZQEIITELKPTW4TISZWTD6XZGI4UJV7UCLUXAEQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX): protocol: expected piece hash; context canceled; EOF Oct 15 09:48:42.434666138 (524h17m27.896758621s) [noise=true , address=45.91.116.5:4299 , piece_id=4ARMXXE5LBRE2OZQEIITELKPTW4TISZWTD6XZGI4UJV7UCLUXAEQ, seg_pos={0 0}, upload=461218]: Long tail cancellation due to optimal threshold Oct 15 09:48:42.434692849 (524h17m27.896785432s) [noise=true , address=45.91.116.12:4226 , piece_id=DRXRGO7OXEKVS5ACL6FC7RKLGBBD6KNI7X6E5DM5D5VV5PEKN7FQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1): protocol: expected piece hash; context canceled; EOF Oct 15 09:48:42.434706184 (524h17m27.896798657s) [noise=true , address=45.91.116.12:4226 , piece_id=DRXRGO7OXEKVS5ACL6FC7RKLGBBD6KNI7X6E5DM5D5VV5PEKN7FQ, seg_pos={0 0}, upload=461218]: Long tail cancellation due to optimal threshold Oct 15 09:48:42.434718318 (524h17m27.896810911s) [noise=true , address=45.91.116.4:4940 , piece_id=3MCSUDYN73NIVJTGKPOVY6N3MALB3GBXI7I5NAX4HXPDHVMQ52BA, seg_pos={0 0}, upload=461218]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe): protocol: expected piece hash; context canceled; EOF Oct 15 09:48:42.434728447 (524h17m27.89682093s) [noise=true , address=45.91.116.4:4940 , piece_id=3MCSUDYN73NIVJTGKPOVY6N3MALB3GBXI7I5NAX4HXPDHVMQ52BA, seg_pos={0 0}, upload=461218]: Long tail cancellation due to optimal threshold Oct 15 09:48:42.434760458 (524h17m27.896852931s) [seg_pos={0 0}, upload=461218]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 15 09:48:42.434866900 (524h17m27.896959384s) [upload=461218]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 15 09:48:42.434873884 (524h17m27.896966367s) [upload=461218]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 15 09:48:42.760633614 (524h17m28.222726087s) [upload=461218]: Done uploading segments... Oct 15 09:48:42.760655435 (524h17m28.222747918s) [upload=461218]: Upload finished. err: <nil> Duration: 6.711179513s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..............................X...X.........................] duration: 6.060340566s FAIL (2025-10-15 10:05:27 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, Piece ID: VH7ZJKSZ2NUXQJZE2IKNWP5PAL3TN2NNHD3ATZUQHGKKYMSSM3LQ) context deadline exceeded; piecestore: (Node ID: 1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N, Piece ID: UMQRQYPZ7SWUCO2N7JDCEVI5U5P36FDL3SUM6HIYJVWVOZTWNXQQ) context deadline exceeded ======LOGS====== Oct 15 09:48:38.801058027 (524h17m24.26315051s) [upload=461218]: Starting upload Oct 15 09:48:38.801138119 (524h17m24.263230592s) [upload=461218]: Done starting upload Oct 15 09:48:38.801288545 (524h17m24.263381028s) [upload=461218]: Uploading segments... Oct 15 09:48:38.801442509 (524h17m24.263535002s) [upload=461218]: Got next segment. Inline: false Oct 15 09:48:38.801486663 (524h17m24.263579136s) [upload=461218]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 15 09:48:38.801493245 (524h17m24.263585728s) [upload=461218]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 15 09:48:39.066131918 (524h17m24.528224401s) [seg_pos={0 0}, upload=461218]: Begin upload segment... Oct 15 09:48:39.066205549 (524h17m24.528298022s) [seg_pos={0 0}, upload=461218]: Done begin upload segment. Oct 15 09:48:39.066243090 (524h17m24.528335573s) [noise=true , address=45.91.116.5:4299 , piece_id=4ARMXXE5LBRE2OZQEIITELKPTW4TISZWTD6XZGI4UJV7UCLUXAEQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.066254431 (524h17m24.528346915s) [noise=true , address=45.91.116.11:4117 , piece_id=VH7ZJKSZ2NUXQJZE2IKNWP5PAL3TN2NNHD3ATZUQHGKKYMSSM3LQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.066874873 (524h17m24.528967356s) [noise=true , address=145.79.227.7:4294 , piece_id=H4ZNVIXORNO4MTXOVYDQZI4IJE4VXHZWC6JI35YKOR3UESSXJNTQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.067301676 (524h17m24.529394159s) [noise=true , address=45.91.116.9:4262 , piece_id=X53JZGRLZLX53UZLLCXXUQCFSQD2FO76X74OM45G26HNDU55UXXQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.067705735 (524h17m24.529798218s) [noise=true , address=145.79.227.4:4187 , piece_id=TQVPL6ZA6LHKAJVL6XWSYASZD3MCKKV3553CRFD72XWTVWY62STQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.069183369 (524h17m24.531275862s) [noise=true , address=145.79.227.5:4222 , piece_id=C2JSPN2QVSGSXG6ZW3ZMOFX5YFNN65XY7E62N5OEHOOMK3NSO4KQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.070534782 (524h17m24.532627265s) [noise=true , address=77.90.31.11:4555 , piece_id=U5RYZSD643KSUI4XLL26EAN6FQRPMG62GQMMCQRPEEG2XKSNBIZQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.072356982 (524h17m24.534449466s) [noise=true , address=145.79.227.11:4730 , piece_id=VO2DSM5LT43QAFPWOY3W3PVEJ44GZJWTCWG5QRQQFB363DFVFZDA, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.073961167 (524h17m24.53605364s) [noise=true , address=45.91.116.4:4940 , piece_id=3MCSUDYN73NIVJTGKPOVY6N3MALB3GBXI7I5NAX4HXPDHVMQ52BA, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.075716960 (524h17m24.537809433s) [noise=true , address=145.79.227.16:4910 , piece_id=BIHEJDH7RPI4UCPUNV453QHSEVCLA54PSNNQB2II37XZDRXFZNWQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.079266419 (524h17m24.541358902s) [noise=true , address=145.79.227.14:4838 , piece_id=6K6A4CNEHGB3X6DEBLF5KHLUT4GE6BQBW6IE3RHBML2NWPN255AA, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.080943032 (524h17m24.543035505s) [noise=true , address=45.91.116.12:4226 , piece_id=DRXRGO7OXEKVS5ACL6FC7RKLGBBD6KNI7X6E5DM5D5VV5PEKN7FQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.081340479 (524h17m24.543432962s) [noise=true , address=45.91.116.8:4045 , piece_id=EDXBNLKKRZMT6MKK2KOIZM7DDKA3IHB6UMCXX4YKZF7JY2T3S5CQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.083616223 (524h17m24.545708706s) [noise=true , address=145.79.227.14:4840 , piece_id=B4ZVPRMVH7Y3TCWCLPO7DWRVMJL7VSI73CIT4PEHCMTIEURQKB2Q, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.083633946 (524h17m24.545726429s) [noise=true , address=45.91.116.14:4189 , piece_id=UMQRQYPZ7SWUCO2N7JDCEVI5U5P36FDL3SUM6HIYJVWVOZTWNXQQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.085493247 (524h17m24.54758573s) [noise=true , address=145.79.227.17:4969 , piece_id=R5HIP33QW5HLICARFWDUOW7I75YTDHZTJCIML5X52VKHLPL3J4YQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:40.727263388 (524h17m26.189355871s) [upload=461218]: Next returned nil segment Oct 15 09:48:40.727310076 (524h17m26.189402549s) [upload=461218]: Waiting for error group managing segments... Oct 15 09:48:41.004514306 (524h17m26.466606789s) [noise=true , address=145.79.227.7:4294 , piece_id=H4ZNVIXORNO4MTXOVYDQZI4IJE4VXHZWC6JI35YKOR3UESSXJNTQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.006116557 (524h17m26.46820904s) [noise=true , address=145.79.227.5:4222 , piece_id=C2JSPN2QVSGSXG6ZW3ZMOFX5YFNN65XY7E62N5OEHOOMK3NSO4KQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.012963184 (524h17m26.475055667s) [noise=true , address=145.79.227.16:4910 , piece_id=BIHEJDH7RPI4UCPUNV453QHSEVCLA54PSNNQB2II37XZDRXFZNWQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.012994684 (524h17m26.475087167s) [noise=true , address=145.79.227.14:4840 , piece_id=B4ZVPRMVH7Y3TCWCLPO7DWRVMJL7VSI73CIT4PEHCMTIEURQKB2Q, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.016649956 (524h17m26.478742439s) [noise=true , address=145.79.227.11:4730 , piece_id=VO2DSM5LT43QAFPWOY3W3PVEJ44GZJWTCWG5QRQQFB363DFVFZDA, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.019248864 (524h17m26.481341347s) [noise=true , address=145.79.227.17:4969 , piece_id=R5HIP33QW5HLICARFWDUOW7I75YTDHZTJCIML5X52VKHLPL3J4YQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.075991585 (524h17m26.538084068s) [noise=true , address=145.79.227.4:4187 , piece_id=TQVPL6ZA6LHKAJVL6XWSYASZD3MCKKV3553CRFD72XWTVWY62STQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.118655146 (524h17m26.580747619s) [noise=true , address=145.79.227.14:4838 , piece_id=6K6A4CNEHGB3X6DEBLF5KHLUT4GE6BQBW6IE3RHBML2NWPN255AA, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.285934795 (524h17m26.748027278s) [noise=true , address=77.90.31.11:4555 , piece_id=U5RYZSD643KSUI4XLL26EAN6FQRPMG62GQMMCQRPEEG2XKSNBIZQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.744525465 (524h17m27.206617948s) [noise=true , address=45.91.116.9:4262 , piece_id=X53JZGRLZLX53UZLLCXXUQCFSQD2FO76X74OM45G26HNDU55UXXQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:42.218961989 (524h17m27.681054462s) [noise=true , address=45.91.116.11:4117 , piece_id=VH7ZJKSZ2NUXQJZE2IKNWP5PAL3TN2NNHD3ATZUQHGKKYMSSM3LQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:42.313638675 (524h17m27.775731158s) [noise=true , address=45.91.116.14:4189 , piece_id=UMQRQYPZ7SWUCO2N7JDCEVI5U5P36FDL3SUM6HIYJVWVOZTWNXQQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:42.434283559 (524h17m27.896376053s) [noise=true , address=45.91.116.8:4045 , piece_id=EDXBNLKKRZMT6MKK2KOIZM7DDKA3IHB6UMCXX4YKZF7JY2T3S5CQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:42.434321832 (524h17m27.896414305s) [seg_pos={0 0}, upload=461218]: Segment reached optimal threshold of 13 pieces. Oct 15 09:48:42.434605041 (524h17m27.896697534s) [noise=true , address=45.91.116.5:4299 , piece_id=4ARMXXE5LBRE2OZQEIITELKPTW4TISZWTD6XZGI4UJV7UCLUXAEQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX): protocol: expected piece hash; context canceled; EOF Oct 15 09:48:42.434666138 (524h17m27.896758621s) [noise=true , address=45.91.116.5:4299 , piece_id=4ARMXXE5LBRE2OZQEIITELKPTW4TISZWTD6XZGI4UJV7UCLUXAEQ, seg_pos={0 0}, upload=461218]: Long tail cancellation due to optimal threshold Oct 15 09:48:42.434692849 (524h17m27.896785432s) [noise=true , address=45.91.116.12:4226 , piece_id=DRXRGO7OXEKVS5ACL6FC7RKLGBBD6KNI7X6E5DM5D5VV5PEKN7FQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1): protocol: expected piece hash; context canceled; EOF Oct 15 09:48:42.434706184 (524h17m27.896798657s) [noise=true , address=45.91.116.12:4226 , piece_id=DRXRGO7OXEKVS5ACL6FC7RKLGBBD6KNI7X6E5DM5D5VV5PEKN7FQ, seg_pos={0 0}, upload=461218]: Long tail cancellation due to optimal threshold Oct 15 09:48:42.434718318 (524h17m27.896810911s) [noise=true , address=45.91.116.4:4940 , piece_id=3MCSUDYN73NIVJTGKPOVY6N3MALB3GBXI7I5NAX4HXPDHVMQ52BA, seg_pos={0 0}, upload=461218]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe): protocol: expected piece hash; context canceled; EOF Oct 15 09:48:42.434728447 (524h17m27.89682093s) [noise=true , address=45.91.116.4:4940 , piece_id=3MCSUDYN73NIVJTGKPOVY6N3MALB3GBXI7I5NAX4HXPDHVMQ52BA, seg_pos={0 0}, upload=461218]: Long tail cancellation due to optimal threshold Oct 15 09:48:42.434760458 (524h17m27.896852931s) [seg_pos={0 0}, upload=461218]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 15 09:48:42.434866900 (524h17m27.896959384s) [upload=461218]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 15 09:48:42.434873884 (524h17m27.896966367s) [upload=461218]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 15 09:48:42.760633614 (524h17m28.222726087s) [upload=461218]: Done uploading segments... Oct 15 09:48:42.760655435 (524h17m28.222747918s) [upload=461218]: Upload finished. err: <nil> Duration: 6.060369862s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...............................X...X........................] duration: 6.846876442s FAIL (2025-10-15 10:04:38 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, Piece ID: VH7ZJKSZ2NUXQJZE2IKNWP5PAL3TN2NNHD3ATZUQHGKKYMSSM3LQ) context deadline exceeded; piecestore: (Node ID: 1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N, Piece ID: UMQRQYPZ7SWUCO2N7JDCEVI5U5P36FDL3SUM6HIYJVWVOZTWNXQQ) context deadline exceeded ======LOGS====== Oct 15 09:48:38.801058027 (524h17m24.26315051s) [upload=461218]: Starting upload Oct 15 09:48:38.801138119 (524h17m24.263230592s) [upload=461218]: Done starting upload Oct 15 09:48:38.801288545 (524h17m24.263381028s) [upload=461218]: Uploading segments... Oct 15 09:48:38.801442509 (524h17m24.263535002s) [upload=461218]: Got next segment. Inline: false Oct 15 09:48:38.801486663 (524h17m24.263579136s) [upload=461218]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 15 09:48:38.801493245 (524h17m24.263585728s) [upload=461218]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 15 09:48:39.066131918 (524h17m24.528224401s) [seg_pos={0 0}, upload=461218]: Begin upload segment... Oct 15 09:48:39.066205549 (524h17m24.528298022s) [seg_pos={0 0}, upload=461218]: Done begin upload segment. Oct 15 09:48:39.066243090 (524h17m24.528335573s) [noise=true , address=45.91.116.5:4299 , piece_id=4ARMXXE5LBRE2OZQEIITELKPTW4TISZWTD6XZGI4UJV7UCLUXAEQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.066254431 (524h17m24.528346915s) [noise=true , address=45.91.116.11:4117 , piece_id=VH7ZJKSZ2NUXQJZE2IKNWP5PAL3TN2NNHD3ATZUQHGKKYMSSM3LQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.066874873 (524h17m24.528967356s) [noise=true , address=145.79.227.7:4294 , piece_id=H4ZNVIXORNO4MTXOVYDQZI4IJE4VXHZWC6JI35YKOR3UESSXJNTQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.067301676 (524h17m24.529394159s) [noise=true , address=45.91.116.9:4262 , piece_id=X53JZGRLZLX53UZLLCXXUQCFSQD2FO76X74OM45G26HNDU55UXXQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.067705735 (524h17m24.529798218s) [noise=true , address=145.79.227.4:4187 , piece_id=TQVPL6ZA6LHKAJVL6XWSYASZD3MCKKV3553CRFD72XWTVWY62STQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.069183369 (524h17m24.531275862s) [noise=true , address=145.79.227.5:4222 , piece_id=C2JSPN2QVSGSXG6ZW3ZMOFX5YFNN65XY7E62N5OEHOOMK3NSO4KQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.070534782 (524h17m24.532627265s) [noise=true , address=77.90.31.11:4555 , piece_id=U5RYZSD643KSUI4XLL26EAN6FQRPMG62GQMMCQRPEEG2XKSNBIZQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.072356982 (524h17m24.534449466s) [noise=true , address=145.79.227.11:4730 , piece_id=VO2DSM5LT43QAFPWOY3W3PVEJ44GZJWTCWG5QRQQFB363DFVFZDA, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.073961167 (524h17m24.53605364s) [noise=true , address=45.91.116.4:4940 , piece_id=3MCSUDYN73NIVJTGKPOVY6N3MALB3GBXI7I5NAX4HXPDHVMQ52BA, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.075716960 (524h17m24.537809433s) [noise=true , address=145.79.227.16:4910 , piece_id=BIHEJDH7RPI4UCPUNV453QHSEVCLA54PSNNQB2II37XZDRXFZNWQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.079266419 (524h17m24.541358902s) [noise=true , address=145.79.227.14:4838 , piece_id=6K6A4CNEHGB3X6DEBLF5KHLUT4GE6BQBW6IE3RHBML2NWPN255AA, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.080943032 (524h17m24.543035505s) [noise=true , address=45.91.116.12:4226 , piece_id=DRXRGO7OXEKVS5ACL6FC7RKLGBBD6KNI7X6E5DM5D5VV5PEKN7FQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.081340479 (524h17m24.543432962s) [noise=true , address=45.91.116.8:4045 , piece_id=EDXBNLKKRZMT6MKK2KOIZM7DDKA3IHB6UMCXX4YKZF7JY2T3S5CQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.083616223 (524h17m24.545708706s) [noise=true , address=145.79.227.14:4840 , piece_id=B4ZVPRMVH7Y3TCWCLPO7DWRVMJL7VSI73CIT4PEHCMTIEURQKB2Q, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.083633946 (524h17m24.545726429s) [noise=true , address=45.91.116.14:4189 , piece_id=UMQRQYPZ7SWUCO2N7JDCEVI5U5P36FDL3SUM6HIYJVWVOZTWNXQQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.085493247 (524h17m24.54758573s) [noise=true , address=145.79.227.17:4969 , piece_id=R5HIP33QW5HLICARFWDUOW7I75YTDHZTJCIML5X52VKHLPL3J4YQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:40.727263388 (524h17m26.189355871s) [upload=461218]: Next returned nil segment Oct 15 09:48:40.727310076 (524h17m26.189402549s) [upload=461218]: Waiting for error group managing segments... Oct 15 09:48:41.004514306 (524h17m26.466606789s) [noise=true , address=145.79.227.7:4294 , piece_id=H4ZNVIXORNO4MTXOVYDQZI4IJE4VXHZWC6JI35YKOR3UESSXJNTQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.006116557 (524h17m26.46820904s) [noise=true , address=145.79.227.5:4222 , piece_id=C2JSPN2QVSGSXG6ZW3ZMOFX5YFNN65XY7E62N5OEHOOMK3NSO4KQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.012963184 (524h17m26.475055667s) [noise=true , address=145.79.227.16:4910 , piece_id=BIHEJDH7RPI4UCPUNV453QHSEVCLA54PSNNQB2II37XZDRXFZNWQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.012994684 (524h17m26.475087167s) [noise=true , address=145.79.227.14:4840 , piece_id=B4ZVPRMVH7Y3TCWCLPO7DWRVMJL7VSI73CIT4PEHCMTIEURQKB2Q, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.016649956 (524h17m26.478742439s) [noise=true , address=145.79.227.11:4730 , piece_id=VO2DSM5LT43QAFPWOY3W3PVEJ44GZJWTCWG5QRQQFB363DFVFZDA, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.019248864 (524h17m26.481341347s) [noise=true , address=145.79.227.17:4969 , piece_id=R5HIP33QW5HLICARFWDUOW7I75YTDHZTJCIML5X52VKHLPL3J4YQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.075991585 (524h17m26.538084068s) [noise=true , address=145.79.227.4:4187 , piece_id=TQVPL6ZA6LHKAJVL6XWSYASZD3MCKKV3553CRFD72XWTVWY62STQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.118655146 (524h17m26.580747619s) [noise=true , address=145.79.227.14:4838 , piece_id=6K6A4CNEHGB3X6DEBLF5KHLUT4GE6BQBW6IE3RHBML2NWPN255AA, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.285934795 (524h17m26.748027278s) [noise=true , address=77.90.31.11:4555 , piece_id=U5RYZSD643KSUI4XLL26EAN6FQRPMG62GQMMCQRPEEG2XKSNBIZQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.744525465 (524h17m27.206617948s) [noise=true , address=45.91.116.9:4262 , piece_id=X53JZGRLZLX53UZLLCXXUQCFSQD2FO76X74OM45G26HNDU55UXXQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:42.218961989 (524h17m27.681054462s) [noise=true , address=45.91.116.11:4117 , piece_id=VH7ZJKSZ2NUXQJZE2IKNWP5PAL3TN2NNHD3ATZUQHGKKYMSSM3LQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:42.313638675 (524h17m27.775731158s) [noise=true , address=45.91.116.14:4189 , piece_id=UMQRQYPZ7SWUCO2N7JDCEVI5U5P36FDL3SUM6HIYJVWVOZTWNXQQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:42.434283559 (524h17m27.896376053s) [noise=true , address=45.91.116.8:4045 , piece_id=EDXBNLKKRZMT6MKK2KOIZM7DDKA3IHB6UMCXX4YKZF7JY2T3S5CQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:42.434321832 (524h17m27.896414305s) [seg_pos={0 0}, upload=461218]: Segment reached optimal threshold of 13 pieces. Oct 15 09:48:42.434605041 (524h17m27.896697534s) [noise=true , address=45.91.116.5:4299 , piece_id=4ARMXXE5LBRE2OZQEIITELKPTW4TISZWTD6XZGI4UJV7UCLUXAEQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX): protocol: expected piece hash; context canceled; EOF Oct 15 09:48:42.434666138 (524h17m27.896758621s) [noise=true , address=45.91.116.5:4299 , piece_id=4ARMXXE5LBRE2OZQEIITELKPTW4TISZWTD6XZGI4UJV7UCLUXAEQ, seg_pos={0 0}, upload=461218]: Long tail cancellation due to optimal threshold Oct 15 09:48:42.434692849 (524h17m27.896785432s) [noise=true , address=45.91.116.12:4226 , piece_id=DRXRGO7OXEKVS5ACL6FC7RKLGBBD6KNI7X6E5DM5D5VV5PEKN7FQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1): protocol: expected piece hash; context canceled; EOF Oct 15 09:48:42.434706184 (524h17m27.896798657s) [noise=true , address=45.91.116.12:4226 , piece_id=DRXRGO7OXEKVS5ACL6FC7RKLGBBD6KNI7X6E5DM5D5VV5PEKN7FQ, seg_pos={0 0}, upload=461218]: Long tail cancellation due to optimal threshold Oct 15 09:48:42.434718318 (524h17m27.896810911s) [noise=true , address=45.91.116.4:4940 , piece_id=3MCSUDYN73NIVJTGKPOVY6N3MALB3GBXI7I5NAX4HXPDHVMQ52BA, seg_pos={0 0}, upload=461218]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe): protocol: expected piece hash; context canceled; EOF Oct 15 09:48:42.434728447 (524h17m27.89682093s) [noise=true , address=45.91.116.4:4940 , piece_id=3MCSUDYN73NIVJTGKPOVY6N3MALB3GBXI7I5NAX4HXPDHVMQ52BA, seg_pos={0 0}, upload=461218]: Long tail cancellation due to optimal threshold Oct 15 09:48:42.434760458 (524h17m27.896852931s) [seg_pos={0 0}, upload=461218]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 15 09:48:42.434866900 (524h17m27.896959384s) [upload=461218]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 15 09:48:42.434873884 (524h17m27.896966367s) [upload=461218]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 15 09:48:42.760633614 (524h17m28.222726087s) [upload=461218]: Done uploading segments... Oct 15 09:48:42.760655435 (524h17m28.222747918s) [upload=461218]: Upload finished. err: <nil> Duration: 6.84691248s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [................................X...X.......................] duration: 6.505674495s FAIL (2025-10-15 10:04:09 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, Piece ID: VH7ZJKSZ2NUXQJZE2IKNWP5PAL3TN2NNHD3ATZUQHGKKYMSSM3LQ) context deadline exceeded; piecestore: (Node ID: 1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N, Piece ID: UMQRQYPZ7SWUCO2N7JDCEVI5U5P36FDL3SUM6HIYJVWVOZTWNXQQ) context deadline exceeded ======LOGS====== Oct 15 09:48:38.801058027 (524h17m24.26315051s) [upload=461218]: Starting upload Oct 15 09:48:38.801138119 (524h17m24.263230592s) [upload=461218]: Done starting upload Oct 15 09:48:38.801288545 (524h17m24.263381028s) [upload=461218]: Uploading segments... Oct 15 09:48:38.801442509 (524h17m24.263535002s) [upload=461218]: Got next segment. Inline: false Oct 15 09:48:38.801486663 (524h17m24.263579136s) [upload=461218]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 15 09:48:38.801493245 (524h17m24.263585728s) [upload=461218]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 15 09:48:39.066131918 (524h17m24.528224401s) [seg_pos={0 0}, upload=461218]: Begin upload segment... Oct 15 09:48:39.066205549 (524h17m24.528298022s) [seg_pos={0 0}, upload=461218]: Done begin upload segment. Oct 15 09:48:39.066243090 (524h17m24.528335573s) [noise=true , address=45.91.116.5:4299 , piece_id=4ARMXXE5LBRE2OZQEIITELKPTW4TISZWTD6XZGI4UJV7UCLUXAEQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.066254431 (524h17m24.528346915s) [noise=true , address=45.91.116.11:4117 , piece_id=VH7ZJKSZ2NUXQJZE2IKNWP5PAL3TN2NNHD3ATZUQHGKKYMSSM3LQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.066874873 (524h17m24.528967356s) [noise=true , address=145.79.227.7:4294 , piece_id=H4ZNVIXORNO4MTXOVYDQZI4IJE4VXHZWC6JI35YKOR3UESSXJNTQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.067301676 (524h17m24.529394159s) [noise=true , address=45.91.116.9:4262 , piece_id=X53JZGRLZLX53UZLLCXXUQCFSQD2FO76X74OM45G26HNDU55UXXQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.067705735 (524h17m24.529798218s) [noise=true , address=145.79.227.4:4187 , piece_id=TQVPL6ZA6LHKAJVL6XWSYASZD3MCKKV3553CRFD72XWTVWY62STQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.069183369 (524h17m24.531275862s) [noise=true , address=145.79.227.5:4222 , piece_id=C2JSPN2QVSGSXG6ZW3ZMOFX5YFNN65XY7E62N5OEHOOMK3NSO4KQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.070534782 (524h17m24.532627265s) [noise=true , address=77.90.31.11:4555 , piece_id=U5RYZSD643KSUI4XLL26EAN6FQRPMG62GQMMCQRPEEG2XKSNBIZQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.072356982 (524h17m24.534449466s) [noise=true , address=145.79.227.11:4730 , piece_id=VO2DSM5LT43QAFPWOY3W3PVEJ44GZJWTCWG5QRQQFB363DFVFZDA, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.073961167 (524h17m24.53605364s) [noise=true , address=45.91.116.4:4940 , piece_id=3MCSUDYN73NIVJTGKPOVY6N3MALB3GBXI7I5NAX4HXPDHVMQ52BA, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.075716960 (524h17m24.537809433s) [noise=true , address=145.79.227.16:4910 , piece_id=BIHEJDH7RPI4UCPUNV453QHSEVCLA54PSNNQB2II37XZDRXFZNWQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.079266419 (524h17m24.541358902s) [noise=true , address=145.79.227.14:4838 , piece_id=6K6A4CNEHGB3X6DEBLF5KHLUT4GE6BQBW6IE3RHBML2NWPN255AA, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.080943032 (524h17m24.543035505s) [noise=true , address=45.91.116.12:4226 , piece_id=DRXRGO7OXEKVS5ACL6FC7RKLGBBD6KNI7X6E5DM5D5VV5PEKN7FQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.081340479 (524h17m24.543432962s) [noise=true , address=45.91.116.8:4045 , piece_id=EDXBNLKKRZMT6MKK2KOIZM7DDKA3IHB6UMCXX4YKZF7JY2T3S5CQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.083616223 (524h17m24.545708706s) [noise=true , address=145.79.227.14:4840 , piece_id=B4ZVPRMVH7Y3TCWCLPO7DWRVMJL7VSI73CIT4PEHCMTIEURQKB2Q, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.083633946 (524h17m24.545726429s) [noise=true , address=45.91.116.14:4189 , piece_id=UMQRQYPZ7SWUCO2N7JDCEVI5U5P36FDL3SUM6HIYJVWVOZTWNXQQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.085493247 (524h17m24.54758573s) [noise=true , address=145.79.227.17:4969 , piece_id=R5HIP33QW5HLICARFWDUOW7I75YTDHZTJCIML5X52VKHLPL3J4YQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:40.727263388 (524h17m26.189355871s) [upload=461218]: Next returned nil segment Oct 15 09:48:40.727310076 (524h17m26.189402549s) [upload=461218]: Waiting for error group managing segments... Oct 15 09:48:41.004514306 (524h17m26.466606789s) [noise=true , address=145.79.227.7:4294 , piece_id=H4ZNVIXORNO4MTXOVYDQZI4IJE4VXHZWC6JI35YKOR3UESSXJNTQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.006116557 (524h17m26.46820904s) [noise=true , address=145.79.227.5:4222 , piece_id=C2JSPN2QVSGSXG6ZW3ZMOFX5YFNN65XY7E62N5OEHOOMK3NSO4KQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.012963184 (524h17m26.475055667s) [noise=true , address=145.79.227.16:4910 , piece_id=BIHEJDH7RPI4UCPUNV453QHSEVCLA54PSNNQB2II37XZDRXFZNWQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.012994684 (524h17m26.475087167s) [noise=true , address=145.79.227.14:4840 , piece_id=B4ZVPRMVH7Y3TCWCLPO7DWRVMJL7VSI73CIT4PEHCMTIEURQKB2Q, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.016649956 (524h17m26.478742439s) [noise=true , address=145.79.227.11:4730 , piece_id=VO2DSM5LT43QAFPWOY3W3PVEJ44GZJWTCWG5QRQQFB363DFVFZDA, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.019248864 (524h17m26.481341347s) [noise=true , address=145.79.227.17:4969 , piece_id=R5HIP33QW5HLICARFWDUOW7I75YTDHZTJCIML5X52VKHLPL3J4YQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.075991585 (524h17m26.538084068s) [noise=true , address=145.79.227.4:4187 , piece_id=TQVPL6ZA6LHKAJVL6XWSYASZD3MCKKV3553CRFD72XWTVWY62STQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.118655146 (524h17m26.580747619s) [noise=true , address=145.79.227.14:4838 , piece_id=6K6A4CNEHGB3X6DEBLF5KHLUT4GE6BQBW6IE3RHBML2NWPN255AA, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.285934795 (524h17m26.748027278s) [noise=true , address=77.90.31.11:4555 , piece_id=U5RYZSD643KSUI4XLL26EAN6FQRPMG62GQMMCQRPEEG2XKSNBIZQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.744525465 (524h17m27.206617948s) [noise=true , address=45.91.116.9:4262 , piece_id=X53JZGRLZLX53UZLLCXXUQCFSQD2FO76X74OM45G26HNDU55UXXQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:42.218961989 (524h17m27.681054462s) [noise=true , address=45.91.116.11:4117 , piece_id=VH7ZJKSZ2NUXQJZE2IKNWP5PAL3TN2NNHD3ATZUQHGKKYMSSM3LQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:42.313638675 (524h17m27.775731158s) [noise=true , address=45.91.116.14:4189 , piece_id=UMQRQYPZ7SWUCO2N7JDCEVI5U5P36FDL3SUM6HIYJVWVOZTWNXQQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:42.434283559 (524h17m27.896376053s) [noise=true , address=45.91.116.8:4045 , piece_id=EDXBNLKKRZMT6MKK2KOIZM7DDKA3IHB6UMCXX4YKZF7JY2T3S5CQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:42.434321832 (524h17m27.896414305s) [seg_pos={0 0}, upload=461218]: Segment reached optimal threshold of 13 pieces. Oct 15 09:48:42.434605041 (524h17m27.896697534s) [noise=true , address=45.91.116.5:4299 , piece_id=4ARMXXE5LBRE2OZQEIITELKPTW4TISZWTD6XZGI4UJV7UCLUXAEQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX): protocol: expected piece hash; context canceled; EOF Oct 15 09:48:42.434666138 (524h17m27.896758621s) [noise=true , address=45.91.116.5:4299 , piece_id=4ARMXXE5LBRE2OZQEIITELKPTW4TISZWTD6XZGI4UJV7UCLUXAEQ, seg_pos={0 0}, upload=461218]: Long tail cancellation due to optimal threshold Oct 15 09:48:42.434692849 (524h17m27.896785432s) [noise=true , address=45.91.116.12:4226 , piece_id=DRXRGO7OXEKVS5ACL6FC7RKLGBBD6KNI7X6E5DM5D5VV5PEKN7FQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1): protocol: expected piece hash; context canceled; EOF Oct 15 09:48:42.434706184 (524h17m27.896798657s) [noise=true , address=45.91.116.12:4226 , piece_id=DRXRGO7OXEKVS5ACL6FC7RKLGBBD6KNI7X6E5DM5D5VV5PEKN7FQ, seg_pos={0 0}, upload=461218]: Long tail cancellation due to optimal threshold Oct 15 09:48:42.434718318 (524h17m27.896810911s) [noise=true , address=45.91.116.4:4940 , piece_id=3MCSUDYN73NIVJTGKPOVY6N3MALB3GBXI7I5NAX4HXPDHVMQ52BA, seg_pos={0 0}, upload=461218]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe): protocol: expected piece hash; context canceled; EOF Oct 15 09:48:42.434728447 (524h17m27.89682093s) [noise=true , address=45.91.116.4:4940 , piece_id=3MCSUDYN73NIVJTGKPOVY6N3MALB3GBXI7I5NAX4HXPDHVMQ52BA, seg_pos={0 0}, upload=461218]: Long tail cancellation due to optimal threshold Oct 15 09:48:42.434760458 (524h17m27.896852931s) [seg_pos={0 0}, upload=461218]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 15 09:48:42.434866900 (524h17m27.896959384s) [upload=461218]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 15 09:48:42.434873884 (524h17m27.896966367s) [upload=461218]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 15 09:48:42.760633614 (524h17m28.222726087s) [upload=461218]: Done uploading segments... Oct 15 09:48:42.760655435 (524h17m28.222747918s) [upload=461218]: Upload finished. err: <nil> Duration: 6.505723207s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.................................X...X......................] duration: 6.047844542s FAIL (2025-10-15 10:03:21 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, Piece ID: VH7ZJKSZ2NUXQJZE2IKNWP5PAL3TN2NNHD3ATZUQHGKKYMSSM3LQ) context deadline exceeded; piecestore: (Node ID: 1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N, Piece ID: UMQRQYPZ7SWUCO2N7JDCEVI5U5P36FDL3SUM6HIYJVWVOZTWNXQQ) context deadline exceeded ======LOGS====== Oct 15 09:48:38.801058027 (524h17m24.26315051s) [upload=461218]: Starting upload Oct 15 09:48:38.801138119 (524h17m24.263230592s) [upload=461218]: Done starting upload Oct 15 09:48:38.801288545 (524h17m24.263381028s) [upload=461218]: Uploading segments... Oct 15 09:48:38.801442509 (524h17m24.263535002s) [upload=461218]: Got next segment. Inline: false Oct 15 09:48:38.801486663 (524h17m24.263579136s) [upload=461218]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 15 09:48:38.801493245 (524h17m24.263585728s) [upload=461218]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 15 09:48:39.066131918 (524h17m24.528224401s) [seg_pos={0 0}, upload=461218]: Begin upload segment... Oct 15 09:48:39.066205549 (524h17m24.528298022s) [seg_pos={0 0}, upload=461218]: Done begin upload segment. Oct 15 09:48:39.066243090 (524h17m24.528335573s) [noise=true , address=45.91.116.5:4299 , piece_id=4ARMXXE5LBRE2OZQEIITELKPTW4TISZWTD6XZGI4UJV7UCLUXAEQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.066254431 (524h17m24.528346915s) [noise=true , address=45.91.116.11:4117 , piece_id=VH7ZJKSZ2NUXQJZE2IKNWP5PAL3TN2NNHD3ATZUQHGKKYMSSM3LQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.066874873 (524h17m24.528967356s) [noise=true , address=145.79.227.7:4294 , piece_id=H4ZNVIXORNO4MTXOVYDQZI4IJE4VXHZWC6JI35YKOR3UESSXJNTQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.067301676 (524h17m24.529394159s) [noise=true , address=45.91.116.9:4262 , piece_id=X53JZGRLZLX53UZLLCXXUQCFSQD2FO76X74OM45G26HNDU55UXXQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.067705735 (524h17m24.529798218s) [noise=true , address=145.79.227.4:4187 , piece_id=TQVPL6ZA6LHKAJVL6XWSYASZD3MCKKV3553CRFD72XWTVWY62STQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.069183369 (524h17m24.531275862s) [noise=true , address=145.79.227.5:4222 , piece_id=C2JSPN2QVSGSXG6ZW3ZMOFX5YFNN65XY7E62N5OEHOOMK3NSO4KQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.070534782 (524h17m24.532627265s) [noise=true , address=77.90.31.11:4555 , piece_id=U5RYZSD643KSUI4XLL26EAN6FQRPMG62GQMMCQRPEEG2XKSNBIZQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.072356982 (524h17m24.534449466s) [noise=true , address=145.79.227.11:4730 , piece_id=VO2DSM5LT43QAFPWOY3W3PVEJ44GZJWTCWG5QRQQFB363DFVFZDA, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.073961167 (524h17m24.53605364s) [noise=true , address=45.91.116.4:4940 , piece_id=3MCSUDYN73NIVJTGKPOVY6N3MALB3GBXI7I5NAX4HXPDHVMQ52BA, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.075716960 (524h17m24.537809433s) [noise=true , address=145.79.227.16:4910 , piece_id=BIHEJDH7RPI4UCPUNV453QHSEVCLA54PSNNQB2II37XZDRXFZNWQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.079266419 (524h17m24.541358902s) [noise=true , address=145.79.227.14:4838 , piece_id=6K6A4CNEHGB3X6DEBLF5KHLUT4GE6BQBW6IE3RHBML2NWPN255AA, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.080943032 (524h17m24.543035505s) [noise=true , address=45.91.116.12:4226 , piece_id=DRXRGO7OXEKVS5ACL6FC7RKLGBBD6KNI7X6E5DM5D5VV5PEKN7FQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.081340479 (524h17m24.543432962s) [noise=true , address=45.91.116.8:4045 , piece_id=EDXBNLKKRZMT6MKK2KOIZM7DDKA3IHB6UMCXX4YKZF7JY2T3S5CQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.083616223 (524h17m24.545708706s) [noise=true , address=145.79.227.14:4840 , piece_id=B4ZVPRMVH7Y3TCWCLPO7DWRVMJL7VSI73CIT4PEHCMTIEURQKB2Q, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.083633946 (524h17m24.545726429s) [noise=true , address=45.91.116.14:4189 , piece_id=UMQRQYPZ7SWUCO2N7JDCEVI5U5P36FDL3SUM6HIYJVWVOZTWNXQQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.085493247 (524h17m24.54758573s) [noise=true , address=145.79.227.17:4969 , piece_id=R5HIP33QW5HLICARFWDUOW7I75YTDHZTJCIML5X52VKHLPL3J4YQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:40.727263388 (524h17m26.189355871s) [upload=461218]: Next returned nil segment Oct 15 09:48:40.727310076 (524h17m26.189402549s) [upload=461218]: Waiting for error group managing segments... Oct 15 09:48:41.004514306 (524h17m26.466606789s) [noise=true , address=145.79.227.7:4294 , piece_id=H4ZNVIXORNO4MTXOVYDQZI4IJE4VXHZWC6JI35YKOR3UESSXJNTQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.006116557 (524h17m26.46820904s) [noise=true , address=145.79.227.5:4222 , piece_id=C2JSPN2QVSGSXG6ZW3ZMOFX5YFNN65XY7E62N5OEHOOMK3NSO4KQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.012963184 (524h17m26.475055667s) [noise=true , address=145.79.227.16:4910 , piece_id=BIHEJDH7RPI4UCPUNV453QHSEVCLA54PSNNQB2II37XZDRXFZNWQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.012994684 (524h17m26.475087167s) [noise=true , address=145.79.227.14:4840 , piece_id=B4ZVPRMVH7Y3TCWCLPO7DWRVMJL7VSI73CIT4PEHCMTIEURQKB2Q, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.016649956 (524h17m26.478742439s) [noise=true , address=145.79.227.11:4730 , piece_id=VO2DSM5LT43QAFPWOY3W3PVEJ44GZJWTCWG5QRQQFB363DFVFZDA, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.019248864 (524h17m26.481341347s) [noise=true , address=145.79.227.17:4969 , piece_id=R5HIP33QW5HLICARFWDUOW7I75YTDHZTJCIML5X52VKHLPL3J4YQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.075991585 (524h17m26.538084068s) [noise=true , address=145.79.227.4:4187 , piece_id=TQVPL6ZA6LHKAJVL6XWSYASZD3MCKKV3553CRFD72XWTVWY62STQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.118655146 (524h17m26.580747619s) [noise=true , address=145.79.227.14:4838 , piece_id=6K6A4CNEHGB3X6DEBLF5KHLUT4GE6BQBW6IE3RHBML2NWPN255AA, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.285934795 (524h17m26.748027278s) [noise=true , address=77.90.31.11:4555 , piece_id=U5RYZSD643KSUI4XLL26EAN6FQRPMG62GQMMCQRPEEG2XKSNBIZQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.744525465 (524h17m27.206617948s) [noise=true , address=45.91.116.9:4262 , piece_id=X53JZGRLZLX53UZLLCXXUQCFSQD2FO76X74OM45G26HNDU55UXXQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:42.218961989 (524h17m27.681054462s) [noise=true , address=45.91.116.11:4117 , piece_id=VH7ZJKSZ2NUXQJZE2IKNWP5PAL3TN2NNHD3ATZUQHGKKYMSSM3LQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:42.313638675 (524h17m27.775731158s) [noise=true , address=45.91.116.14:4189 , piece_id=UMQRQYPZ7SWUCO2N7JDCEVI5U5P36FDL3SUM6HIYJVWVOZTWNXQQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:42.434283559 (524h17m27.896376053s) [noise=true , address=45.91.116.8:4045 , piece_id=EDXBNLKKRZMT6MKK2KOIZM7DDKA3IHB6UMCXX4YKZF7JY2T3S5CQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:42.434321832 (524h17m27.896414305s) [seg_pos={0 0}, upload=461218]: Segment reached optimal threshold of 13 pieces. Oct 15 09:48:42.434605041 (524h17m27.896697534s) [noise=true , address=45.91.116.5:4299 , piece_id=4ARMXXE5LBRE2OZQEIITELKPTW4TISZWTD6XZGI4UJV7UCLUXAEQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX): protocol: expected piece hash; context canceled; EOF Oct 15 09:48:42.434666138 (524h17m27.896758621s) [noise=true , address=45.91.116.5:4299 , piece_id=4ARMXXE5LBRE2OZQEIITELKPTW4TISZWTD6XZGI4UJV7UCLUXAEQ, seg_pos={0 0}, upload=461218]: Long tail cancellation due to optimal threshold Oct 15 09:48:42.434692849 (524h17m27.896785432s) [noise=true , address=45.91.116.12:4226 , piece_id=DRXRGO7OXEKVS5ACL6FC7RKLGBBD6KNI7X6E5DM5D5VV5PEKN7FQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1): protocol: expected piece hash; context canceled; EOF Oct 15 09:48:42.434706184 (524h17m27.896798657s) [noise=true , address=45.91.116.12:4226 , piece_id=DRXRGO7OXEKVS5ACL6FC7RKLGBBD6KNI7X6E5DM5D5VV5PEKN7FQ, seg_pos={0 0}, upload=461218]: Long tail cancellation due to optimal threshold Oct 15 09:48:42.434718318 (524h17m27.896810911s) [noise=true , address=45.91.116.4:4940 , piece_id=3MCSUDYN73NIVJTGKPOVY6N3MALB3GBXI7I5NAX4HXPDHVMQ52BA, seg_pos={0 0}, upload=461218]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe): protocol: expected piece hash; context canceled; EOF Oct 15 09:48:42.434728447 (524h17m27.89682093s) [noise=true , address=45.91.116.4:4940 , piece_id=3MCSUDYN73NIVJTGKPOVY6N3MALB3GBXI7I5NAX4HXPDHVMQ52BA, seg_pos={0 0}, upload=461218]: Long tail cancellation due to optimal threshold Oct 15 09:48:42.434760458 (524h17m27.896852931s) [seg_pos={0 0}, upload=461218]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 15 09:48:42.434866900 (524h17m27.896959384s) [upload=461218]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 15 09:48:42.434873884 (524h17m27.896966367s) [upload=461218]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 15 09:48:42.760633614 (524h17m28.222726087s) [upload=461218]: Done uploading segments... Oct 15 09:48:42.760655435 (524h17m28.222747918s) [upload=461218]: Upload finished. err: <nil> Duration: 6.047880521s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..................................X...X.....................] duration: 6.067557063s FAIL (2025-10-15 10:02:09 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, Piece ID: VH7ZJKSZ2NUXQJZE2IKNWP5PAL3TN2NNHD3ATZUQHGKKYMSSM3LQ) context deadline exceeded; piecestore: (Node ID: 1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N, Piece ID: UMQRQYPZ7SWUCO2N7JDCEVI5U5P36FDL3SUM6HIYJVWVOZTWNXQQ) context deadline exceeded ======LOGS====== Oct 15 09:48:38.801058027 (524h17m24.26315051s) [upload=461218]: Starting upload Oct 15 09:48:38.801138119 (524h17m24.263230592s) [upload=461218]: Done starting upload Oct 15 09:48:38.801288545 (524h17m24.263381028s) [upload=461218]: Uploading segments... Oct 15 09:48:38.801442509 (524h17m24.263535002s) [upload=461218]: Got next segment. Inline: false Oct 15 09:48:38.801486663 (524h17m24.263579136s) [upload=461218]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 15 09:48:38.801493245 (524h17m24.263585728s) [upload=461218]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 15 09:48:39.066131918 (524h17m24.528224401s) [seg_pos={0 0}, upload=461218]: Begin upload segment... Oct 15 09:48:39.066205549 (524h17m24.528298022s) [seg_pos={0 0}, upload=461218]: Done begin upload segment. Oct 15 09:48:39.066243090 (524h17m24.528335573s) [noise=true , address=45.91.116.5:4299 , piece_id=4ARMXXE5LBRE2OZQEIITELKPTW4TISZWTD6XZGI4UJV7UCLUXAEQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.066254431 (524h17m24.528346915s) [noise=true , address=45.91.116.11:4117 , piece_id=VH7ZJKSZ2NUXQJZE2IKNWP5PAL3TN2NNHD3ATZUQHGKKYMSSM3LQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.066874873 (524h17m24.528967356s) [noise=true , address=145.79.227.7:4294 , piece_id=H4ZNVIXORNO4MTXOVYDQZI4IJE4VXHZWC6JI35YKOR3UESSXJNTQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.067301676 (524h17m24.529394159s) [noise=true , address=45.91.116.9:4262 , piece_id=X53JZGRLZLX53UZLLCXXUQCFSQD2FO76X74OM45G26HNDU55UXXQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.067705735 (524h17m24.529798218s) [noise=true , address=145.79.227.4:4187 , piece_id=TQVPL6ZA6LHKAJVL6XWSYASZD3MCKKV3553CRFD72XWTVWY62STQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.069183369 (524h17m24.531275862s) [noise=true , address=145.79.227.5:4222 , piece_id=C2JSPN2QVSGSXG6ZW3ZMOFX5YFNN65XY7E62N5OEHOOMK3NSO4KQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.070534782 (524h17m24.532627265s) [noise=true , address=77.90.31.11:4555 , piece_id=U5RYZSD643KSUI4XLL26EAN6FQRPMG62GQMMCQRPEEG2XKSNBIZQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.072356982 (524h17m24.534449466s) [noise=true , address=145.79.227.11:4730 , piece_id=VO2DSM5LT43QAFPWOY3W3PVEJ44GZJWTCWG5QRQQFB363DFVFZDA, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.073961167 (524h17m24.53605364s) [noise=true , address=45.91.116.4:4940 , piece_id=3MCSUDYN73NIVJTGKPOVY6N3MALB3GBXI7I5NAX4HXPDHVMQ52BA, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.075716960 (524h17m24.537809433s) [noise=true , address=145.79.227.16:4910 , piece_id=BIHEJDH7RPI4UCPUNV453QHSEVCLA54PSNNQB2II37XZDRXFZNWQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.079266419 (524h17m24.541358902s) [noise=true , address=145.79.227.14:4838 , piece_id=6K6A4CNEHGB3X6DEBLF5KHLUT4GE6BQBW6IE3RHBML2NWPN255AA, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.080943032 (524h17m24.543035505s) [noise=true , address=45.91.116.12:4226 , piece_id=DRXRGO7OXEKVS5ACL6FC7RKLGBBD6KNI7X6E5DM5D5VV5PEKN7FQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.081340479 (524h17m24.543432962s) [noise=true , address=45.91.116.8:4045 , piece_id=EDXBNLKKRZMT6MKK2KOIZM7DDKA3IHB6UMCXX4YKZF7JY2T3S5CQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.083616223 (524h17m24.545708706s) [noise=true , address=145.79.227.14:4840 , piece_id=B4ZVPRMVH7Y3TCWCLPO7DWRVMJL7VSI73CIT4PEHCMTIEURQKB2Q, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.083633946 (524h17m24.545726429s) [noise=true , address=45.91.116.14:4189 , piece_id=UMQRQYPZ7SWUCO2N7JDCEVI5U5P36FDL3SUM6HIYJVWVOZTWNXQQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.085493247 (524h17m24.54758573s) [noise=true , address=145.79.227.17:4969 , piece_id=R5HIP33QW5HLICARFWDUOW7I75YTDHZTJCIML5X52VKHLPL3J4YQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:40.727263388 (524h17m26.189355871s) [upload=461218]: Next returned nil segment Oct 15 09:48:40.727310076 (524h17m26.189402549s) [upload=461218]: Waiting for error group managing segments... Oct 15 09:48:41.004514306 (524h17m26.466606789s) [noise=true , address=145.79.227.7:4294 , piece_id=H4ZNVIXORNO4MTXOVYDQZI4IJE4VXHZWC6JI35YKOR3UESSXJNTQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.006116557 (524h17m26.46820904s) [noise=true , address=145.79.227.5:4222 , piece_id=C2JSPN2QVSGSXG6ZW3ZMOFX5YFNN65XY7E62N5OEHOOMK3NSO4KQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.012963184 (524h17m26.475055667s) [noise=true , address=145.79.227.16:4910 , piece_id=BIHEJDH7RPI4UCPUNV453QHSEVCLA54PSNNQB2II37XZDRXFZNWQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.012994684 (524h17m26.475087167s) [noise=true , address=145.79.227.14:4840 , piece_id=B4ZVPRMVH7Y3TCWCLPO7DWRVMJL7VSI73CIT4PEHCMTIEURQKB2Q, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.016649956 (524h17m26.478742439s) [noise=true , address=145.79.227.11:4730 , piece_id=VO2DSM5LT43QAFPWOY3W3PVEJ44GZJWTCWG5QRQQFB363DFVFZDA, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.019248864 (524h17m26.481341347s) [noise=true , address=145.79.227.17:4969 , piece_id=R5HIP33QW5HLICARFWDUOW7I75YTDHZTJCIML5X52VKHLPL3J4YQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.075991585 (524h17m26.538084068s) [noise=true , address=145.79.227.4:4187 , piece_id=TQVPL6ZA6LHKAJVL6XWSYASZD3MCKKV3553CRFD72XWTVWY62STQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.118655146 (524h17m26.580747619s) [noise=true , address=145.79.227.14:4838 , piece_id=6K6A4CNEHGB3X6DEBLF5KHLUT4GE6BQBW6IE3RHBML2NWPN255AA, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.285934795 (524h17m26.748027278s) [noise=true , address=77.90.31.11:4555 , piece_id=U5RYZSD643KSUI4XLL26EAN6FQRPMG62GQMMCQRPEEG2XKSNBIZQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.744525465 (524h17m27.206617948s) [noise=true , address=45.91.116.9:4262 , piece_id=X53JZGRLZLX53UZLLCXXUQCFSQD2FO76X74OM45G26HNDU55UXXQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:42.218961989 (524h17m27.681054462s) [noise=true , address=45.91.116.11:4117 , piece_id=VH7ZJKSZ2NUXQJZE2IKNWP5PAL3TN2NNHD3ATZUQHGKKYMSSM3LQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:42.313638675 (524h17m27.775731158s) [noise=true , address=45.91.116.14:4189 , piece_id=UMQRQYPZ7SWUCO2N7JDCEVI5U5P36FDL3SUM6HIYJVWVOZTWNXQQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:42.434283559 (524h17m27.896376053s) [noise=true , address=45.91.116.8:4045 , piece_id=EDXBNLKKRZMT6MKK2KOIZM7DDKA3IHB6UMCXX4YKZF7JY2T3S5CQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:42.434321832 (524h17m27.896414305s) [seg_pos={0 0}, upload=461218]: Segment reached optimal threshold of 13 pieces. Oct 15 09:48:42.434605041 (524h17m27.896697534s) [noise=true , address=45.91.116.5:4299 , piece_id=4ARMXXE5LBRE2OZQEIITELKPTW4TISZWTD6XZGI4UJV7UCLUXAEQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX): protocol: expected piece hash; context canceled; EOF Oct 15 09:48:42.434666138 (524h17m27.896758621s) [noise=true , address=45.91.116.5:4299 , piece_id=4ARMXXE5LBRE2OZQEIITELKPTW4TISZWTD6XZGI4UJV7UCLUXAEQ, seg_pos={0 0}, upload=461218]: Long tail cancellation due to optimal threshold Oct 15 09:48:42.434692849 (524h17m27.896785432s) [noise=true , address=45.91.116.12:4226 , piece_id=DRXRGO7OXEKVS5ACL6FC7RKLGBBD6KNI7X6E5DM5D5VV5PEKN7FQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1): protocol: expected piece hash; context canceled; EOF Oct 15 09:48:42.434706184 (524h17m27.896798657s) [noise=true , address=45.91.116.12:4226 , piece_id=DRXRGO7OXEKVS5ACL6FC7RKLGBBD6KNI7X6E5DM5D5VV5PEKN7FQ, seg_pos={0 0}, upload=461218]: Long tail cancellation due to optimal threshold Oct 15 09:48:42.434718318 (524h17m27.896810911s) [noise=true , address=45.91.116.4:4940 , piece_id=3MCSUDYN73NIVJTGKPOVY6N3MALB3GBXI7I5NAX4HXPDHVMQ52BA, seg_pos={0 0}, upload=461218]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe): protocol: expected piece hash; context canceled; EOF Oct 15 09:48:42.434728447 (524h17m27.89682093s) [noise=true , address=45.91.116.4:4940 , piece_id=3MCSUDYN73NIVJTGKPOVY6N3MALB3GBXI7I5NAX4HXPDHVMQ52BA, seg_pos={0 0}, upload=461218]: Long tail cancellation due to optimal threshold Oct 15 09:48:42.434760458 (524h17m27.896852931s) [seg_pos={0 0}, upload=461218]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 15 09:48:42.434866900 (524h17m27.896959384s) [upload=461218]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 15 09:48:42.434873884 (524h17m27.896966367s) [upload=461218]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 15 09:48:42.760633614 (524h17m28.222726087s) [upload=461218]: Done uploading segments... Oct 15 09:48:42.760655435 (524h17m28.222747918s) [upload=461218]: Upload finished. err: <nil> Duration: 6.067589775s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...................................X...X....................] duration: 5.929205415s FAIL (2025-10-15 10:01:21 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, Piece ID: VH7ZJKSZ2NUXQJZE2IKNWP5PAL3TN2NNHD3ATZUQHGKKYMSSM3LQ) context deadline exceeded; piecestore: (Node ID: 1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N, Piece ID: UMQRQYPZ7SWUCO2N7JDCEVI5U5P36FDL3SUM6HIYJVWVOZTWNXQQ) context deadline exceeded ======LOGS====== Oct 15 09:48:38.801058027 (524h17m24.26315051s) [upload=461218]: Starting upload Oct 15 09:48:38.801138119 (524h17m24.263230592s) [upload=461218]: Done starting upload Oct 15 09:48:38.801288545 (524h17m24.263381028s) [upload=461218]: Uploading segments... Oct 15 09:48:38.801442509 (524h17m24.263535002s) [upload=461218]: Got next segment. Inline: false Oct 15 09:48:38.801486663 (524h17m24.263579136s) [upload=461218]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 15 09:48:38.801493245 (524h17m24.263585728s) [upload=461218]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 15 09:48:39.066131918 (524h17m24.528224401s) [seg_pos={0 0}, upload=461218]: Begin upload segment... Oct 15 09:48:39.066205549 (524h17m24.528298022s) [seg_pos={0 0}, upload=461218]: Done begin upload segment. Oct 15 09:48:39.066243090 (524h17m24.528335573s) [noise=true , address=45.91.116.5:4299 , piece_id=4ARMXXE5LBRE2OZQEIITELKPTW4TISZWTD6XZGI4UJV7UCLUXAEQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.066254431 (524h17m24.528346915s) [noise=true , address=45.91.116.11:4117 , piece_id=VH7ZJKSZ2NUXQJZE2IKNWP5PAL3TN2NNHD3ATZUQHGKKYMSSM3LQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.066874873 (524h17m24.528967356s) [noise=true , address=145.79.227.7:4294 , piece_id=H4ZNVIXORNO4MTXOVYDQZI4IJE4VXHZWC6JI35YKOR3UESSXJNTQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.067301676 (524h17m24.529394159s) [noise=true , address=45.91.116.9:4262 , piece_id=X53JZGRLZLX53UZLLCXXUQCFSQD2FO76X74OM45G26HNDU55UXXQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.067705735 (524h17m24.529798218s) [noise=true , address=145.79.227.4:4187 , piece_id=TQVPL6ZA6LHKAJVL6XWSYASZD3MCKKV3553CRFD72XWTVWY62STQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.069183369 (524h17m24.531275862s) [noise=true , address=145.79.227.5:4222 , piece_id=C2JSPN2QVSGSXG6ZW3ZMOFX5YFNN65XY7E62N5OEHOOMK3NSO4KQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.070534782 (524h17m24.532627265s) [noise=true , address=77.90.31.11:4555 , piece_id=U5RYZSD643KSUI4XLL26EAN6FQRPMG62GQMMCQRPEEG2XKSNBIZQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.072356982 (524h17m24.534449466s) [noise=true , address=145.79.227.11:4730 , piece_id=VO2DSM5LT43QAFPWOY3W3PVEJ44GZJWTCWG5QRQQFB363DFVFZDA, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.073961167 (524h17m24.53605364s) [noise=true , address=45.91.116.4:4940 , piece_id=3MCSUDYN73NIVJTGKPOVY6N3MALB3GBXI7I5NAX4HXPDHVMQ52BA, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.075716960 (524h17m24.537809433s) [noise=true , address=145.79.227.16:4910 , piece_id=BIHEJDH7RPI4UCPUNV453QHSEVCLA54PSNNQB2II37XZDRXFZNWQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.079266419 (524h17m24.541358902s) [noise=true , address=145.79.227.14:4838 , piece_id=6K6A4CNEHGB3X6DEBLF5KHLUT4GE6BQBW6IE3RHBML2NWPN255AA, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.080943032 (524h17m24.543035505s) [noise=true , address=45.91.116.12:4226 , piece_id=DRXRGO7OXEKVS5ACL6FC7RKLGBBD6KNI7X6E5DM5D5VV5PEKN7FQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.081340479 (524h17m24.543432962s) [noise=true , address=45.91.116.8:4045 , piece_id=EDXBNLKKRZMT6MKK2KOIZM7DDKA3IHB6UMCXX4YKZF7JY2T3S5CQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.083616223 (524h17m24.545708706s) [noise=true , address=145.79.227.14:4840 , piece_id=B4ZVPRMVH7Y3TCWCLPO7DWRVMJL7VSI73CIT4PEHCMTIEURQKB2Q, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.083633946 (524h17m24.545726429s) [noise=true , address=45.91.116.14:4189 , piece_id=UMQRQYPZ7SWUCO2N7JDCEVI5U5P36FDL3SUM6HIYJVWVOZTWNXQQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.085493247 (524h17m24.54758573s) [noise=true , address=145.79.227.17:4969 , piece_id=R5HIP33QW5HLICARFWDUOW7I75YTDHZTJCIML5X52VKHLPL3J4YQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:40.727263388 (524h17m26.189355871s) [upload=461218]: Next returned nil segment Oct 15 09:48:40.727310076 (524h17m26.189402549s) [upload=461218]: Waiting for error group managing segments... Oct 15 09:48:41.004514306 (524h17m26.466606789s) [noise=true , address=145.79.227.7:4294 , piece_id=H4ZNVIXORNO4MTXOVYDQZI4IJE4VXHZWC6JI35YKOR3UESSXJNTQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.006116557 (524h17m26.46820904s) [noise=true , address=145.79.227.5:4222 , piece_id=C2JSPN2QVSGSXG6ZW3ZMOFX5YFNN65XY7E62N5OEHOOMK3NSO4KQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.012963184 (524h17m26.475055667s) [noise=true , address=145.79.227.16:4910 , piece_id=BIHEJDH7RPI4UCPUNV453QHSEVCLA54PSNNQB2II37XZDRXFZNWQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.012994684 (524h17m26.475087167s) [noise=true , address=145.79.227.14:4840 , piece_id=B4ZVPRMVH7Y3TCWCLPO7DWRVMJL7VSI73CIT4PEHCMTIEURQKB2Q, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.016649956 (524h17m26.478742439s) [noise=true , address=145.79.227.11:4730 , piece_id=VO2DSM5LT43QAFPWOY3W3PVEJ44GZJWTCWG5QRQQFB363DFVFZDA, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.019248864 (524h17m26.481341347s) [noise=true , address=145.79.227.17:4969 , piece_id=R5HIP33QW5HLICARFWDUOW7I75YTDHZTJCIML5X52VKHLPL3J4YQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.075991585 (524h17m26.538084068s) [noise=true , address=145.79.227.4:4187 , piece_id=TQVPL6ZA6LHKAJVL6XWSYASZD3MCKKV3553CRFD72XWTVWY62STQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.118655146 (524h17m26.580747619s) [noise=true , address=145.79.227.14:4838 , piece_id=6K6A4CNEHGB3X6DEBLF5KHLUT4GE6BQBW6IE3RHBML2NWPN255AA, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.285934795 (524h17m26.748027278s) [noise=true , address=77.90.31.11:4555 , piece_id=U5RYZSD643KSUI4XLL26EAN6FQRPMG62GQMMCQRPEEG2XKSNBIZQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.744525465 (524h17m27.206617948s) [noise=true , address=45.91.116.9:4262 , piece_id=X53JZGRLZLX53UZLLCXXUQCFSQD2FO76X74OM45G26HNDU55UXXQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:42.218961989 (524h17m27.681054462s) [noise=true , address=45.91.116.11:4117 , piece_id=VH7ZJKSZ2NUXQJZE2IKNWP5PAL3TN2NNHD3ATZUQHGKKYMSSM3LQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:42.313638675 (524h17m27.775731158s) [noise=true , address=45.91.116.14:4189 , piece_id=UMQRQYPZ7SWUCO2N7JDCEVI5U5P36FDL3SUM6HIYJVWVOZTWNXQQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:42.434283559 (524h17m27.896376053s) [noise=true , address=45.91.116.8:4045 , piece_id=EDXBNLKKRZMT6MKK2KOIZM7DDKA3IHB6UMCXX4YKZF7JY2T3S5CQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:42.434321832 (524h17m27.896414305s) [seg_pos={0 0}, upload=461218]: Segment reached optimal threshold of 13 pieces. Oct 15 09:48:42.434605041 (524h17m27.896697534s) [noise=true , address=45.91.116.5:4299 , piece_id=4ARMXXE5LBRE2OZQEIITELKPTW4TISZWTD6XZGI4UJV7UCLUXAEQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX): protocol: expected piece hash; context canceled; EOF Oct 15 09:48:42.434666138 (524h17m27.896758621s) [noise=true , address=45.91.116.5:4299 , piece_id=4ARMXXE5LBRE2OZQEIITELKPTW4TISZWTD6XZGI4UJV7UCLUXAEQ, seg_pos={0 0}, upload=461218]: Long tail cancellation due to optimal threshold Oct 15 09:48:42.434692849 (524h17m27.896785432s) [noise=true , address=45.91.116.12:4226 , piece_id=DRXRGO7OXEKVS5ACL6FC7RKLGBBD6KNI7X6E5DM5D5VV5PEKN7FQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1): protocol: expected piece hash; context canceled; EOF Oct 15 09:48:42.434706184 (524h17m27.896798657s) [noise=true , address=45.91.116.12:4226 , piece_id=DRXRGO7OXEKVS5ACL6FC7RKLGBBD6KNI7X6E5DM5D5VV5PEKN7FQ, seg_pos={0 0}, upload=461218]: Long tail cancellation due to optimal threshold Oct 15 09:48:42.434718318 (524h17m27.896810911s) [noise=true , address=45.91.116.4:4940 , piece_id=3MCSUDYN73NIVJTGKPOVY6N3MALB3GBXI7I5NAX4HXPDHVMQ52BA, seg_pos={0 0}, upload=461218]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe): protocol: expected piece hash; context canceled; EOF Oct 15 09:48:42.434728447 (524h17m27.89682093s) [noise=true , address=45.91.116.4:4940 , piece_id=3MCSUDYN73NIVJTGKPOVY6N3MALB3GBXI7I5NAX4HXPDHVMQ52BA, seg_pos={0 0}, upload=461218]: Long tail cancellation due to optimal threshold Oct 15 09:48:42.434760458 (524h17m27.896852931s) [seg_pos={0 0}, upload=461218]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 15 09:48:42.434866900 (524h17m27.896959384s) [upload=461218]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 15 09:48:42.434873884 (524h17m27.896966367s) [upload=461218]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 15 09:48:42.760633614 (524h17m28.222726087s) [upload=461218]: Done uploading segments... Oct 15 09:48:42.760655435 (524h17m28.222747918s) [upload=461218]: Upload finished. err: <nil> Duration: 5.929231324s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [....................................X...X...................] duration: 6.035787235s FAIL (2025-10-15 10:00:39 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, Piece ID: VH7ZJKSZ2NUXQJZE2IKNWP5PAL3TN2NNHD3ATZUQHGKKYMSSM3LQ) context deadline exceeded; piecestore: (Node ID: 1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N, Piece ID: UMQRQYPZ7SWUCO2N7JDCEVI5U5P36FDL3SUM6HIYJVWVOZTWNXQQ) context deadline exceeded ======LOGS====== Oct 15 09:48:38.801058027 (524h17m24.26315051s) [upload=461218]: Starting upload Oct 15 09:48:38.801138119 (524h17m24.263230592s) [upload=461218]: Done starting upload Oct 15 09:48:38.801288545 (524h17m24.263381028s) [upload=461218]: Uploading segments... Oct 15 09:48:38.801442509 (524h17m24.263535002s) [upload=461218]: Got next segment. Inline: false Oct 15 09:48:38.801486663 (524h17m24.263579136s) [upload=461218]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 15 09:48:38.801493245 (524h17m24.263585728s) [upload=461218]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 15 09:48:39.066131918 (524h17m24.528224401s) [seg_pos={0 0}, upload=461218]: Begin upload segment... Oct 15 09:48:39.066205549 (524h17m24.528298022s) [seg_pos={0 0}, upload=461218]: Done begin upload segment. Oct 15 09:48:39.066243090 (524h17m24.528335573s) [noise=true , address=45.91.116.5:4299 , piece_id=4ARMXXE5LBRE2OZQEIITELKPTW4TISZWTD6XZGI4UJV7UCLUXAEQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.066254431 (524h17m24.528346915s) [noise=true , address=45.91.116.11:4117 , piece_id=VH7ZJKSZ2NUXQJZE2IKNWP5PAL3TN2NNHD3ATZUQHGKKYMSSM3LQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.066874873 (524h17m24.528967356s) [noise=true , address=145.79.227.7:4294 , piece_id=H4ZNVIXORNO4MTXOVYDQZI4IJE4VXHZWC6JI35YKOR3UESSXJNTQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.067301676 (524h17m24.529394159s) [noise=true , address=45.91.116.9:4262 , piece_id=X53JZGRLZLX53UZLLCXXUQCFSQD2FO76X74OM45G26HNDU55UXXQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.067705735 (524h17m24.529798218s) [noise=true , address=145.79.227.4:4187 , piece_id=TQVPL6ZA6LHKAJVL6XWSYASZD3MCKKV3553CRFD72XWTVWY62STQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.069183369 (524h17m24.531275862s) [noise=true , address=145.79.227.5:4222 , piece_id=C2JSPN2QVSGSXG6ZW3ZMOFX5YFNN65XY7E62N5OEHOOMK3NSO4KQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.070534782 (524h17m24.532627265s) [noise=true , address=77.90.31.11:4555 , piece_id=U5RYZSD643KSUI4XLL26EAN6FQRPMG62GQMMCQRPEEG2XKSNBIZQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.072356982 (524h17m24.534449466s) [noise=true , address=145.79.227.11:4730 , piece_id=VO2DSM5LT43QAFPWOY3W3PVEJ44GZJWTCWG5QRQQFB363DFVFZDA, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.073961167 (524h17m24.53605364s) [noise=true , address=45.91.116.4:4940 , piece_id=3MCSUDYN73NIVJTGKPOVY6N3MALB3GBXI7I5NAX4HXPDHVMQ52BA, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.075716960 (524h17m24.537809433s) [noise=true , address=145.79.227.16:4910 , piece_id=BIHEJDH7RPI4UCPUNV453QHSEVCLA54PSNNQB2II37XZDRXFZNWQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.079266419 (524h17m24.541358902s) [noise=true , address=145.79.227.14:4838 , piece_id=6K6A4CNEHGB3X6DEBLF5KHLUT4GE6BQBW6IE3RHBML2NWPN255AA, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.080943032 (524h17m24.543035505s) [noise=true , address=45.91.116.12:4226 , piece_id=DRXRGO7OXEKVS5ACL6FC7RKLGBBD6KNI7X6E5DM5D5VV5PEKN7FQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.081340479 (524h17m24.543432962s) [noise=true , address=45.91.116.8:4045 , piece_id=EDXBNLKKRZMT6MKK2KOIZM7DDKA3IHB6UMCXX4YKZF7JY2T3S5CQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.083616223 (524h17m24.545708706s) [noise=true , address=145.79.227.14:4840 , piece_id=B4ZVPRMVH7Y3TCWCLPO7DWRVMJL7VSI73CIT4PEHCMTIEURQKB2Q, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.083633946 (524h17m24.545726429s) [noise=true , address=45.91.116.14:4189 , piece_id=UMQRQYPZ7SWUCO2N7JDCEVI5U5P36FDL3SUM6HIYJVWVOZTWNXQQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.085493247 (524h17m24.54758573s) [noise=true , address=145.79.227.17:4969 , piece_id=R5HIP33QW5HLICARFWDUOW7I75YTDHZTJCIML5X52VKHLPL3J4YQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:40.727263388 (524h17m26.189355871s) [upload=461218]: Next returned nil segment Oct 15 09:48:40.727310076 (524h17m26.189402549s) [upload=461218]: Waiting for error group managing segments... Oct 15 09:48:41.004514306 (524h17m26.466606789s) [noise=true , address=145.79.227.7:4294 , piece_id=H4ZNVIXORNO4MTXOVYDQZI4IJE4VXHZWC6JI35YKOR3UESSXJNTQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.006116557 (524h17m26.46820904s) [noise=true , address=145.79.227.5:4222 , piece_id=C2JSPN2QVSGSXG6ZW3ZMOFX5YFNN65XY7E62N5OEHOOMK3NSO4KQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.012963184 (524h17m26.475055667s) [noise=true , address=145.79.227.16:4910 , piece_id=BIHEJDH7RPI4UCPUNV453QHSEVCLA54PSNNQB2II37XZDRXFZNWQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.012994684 (524h17m26.475087167s) [noise=true , address=145.79.227.14:4840 , piece_id=B4ZVPRMVH7Y3TCWCLPO7DWRVMJL7VSI73CIT4PEHCMTIEURQKB2Q, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.016649956 (524h17m26.478742439s) [noise=true , address=145.79.227.11:4730 , piece_id=VO2DSM5LT43QAFPWOY3W3PVEJ44GZJWTCWG5QRQQFB363DFVFZDA, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.019248864 (524h17m26.481341347s) [noise=true , address=145.79.227.17:4969 , piece_id=R5HIP33QW5HLICARFWDUOW7I75YTDHZTJCIML5X52VKHLPL3J4YQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.075991585 (524h17m26.538084068s) [noise=true , address=145.79.227.4:4187 , piece_id=TQVPL6ZA6LHKAJVL6XWSYASZD3MCKKV3553CRFD72XWTVWY62STQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.118655146 (524h17m26.580747619s) [noise=true , address=145.79.227.14:4838 , piece_id=6K6A4CNEHGB3X6DEBLF5KHLUT4GE6BQBW6IE3RHBML2NWPN255AA, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.285934795 (524h17m26.748027278s) [noise=true , address=77.90.31.11:4555 , piece_id=U5RYZSD643KSUI4XLL26EAN6FQRPMG62GQMMCQRPEEG2XKSNBIZQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.744525465 (524h17m27.206617948s) [noise=true , address=45.91.116.9:4262 , piece_id=X53JZGRLZLX53UZLLCXXUQCFSQD2FO76X74OM45G26HNDU55UXXQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:42.218961989 (524h17m27.681054462s) [noise=true , address=45.91.116.11:4117 , piece_id=VH7ZJKSZ2NUXQJZE2IKNWP5PAL3TN2NNHD3ATZUQHGKKYMSSM3LQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:42.313638675 (524h17m27.775731158s) [noise=true , address=45.91.116.14:4189 , piece_id=UMQRQYPZ7SWUCO2N7JDCEVI5U5P36FDL3SUM6HIYJVWVOZTWNXQQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:42.434283559 (524h17m27.896376053s) [noise=true , address=45.91.116.8:4045 , piece_id=EDXBNLKKRZMT6MKK2KOIZM7DDKA3IHB6UMCXX4YKZF7JY2T3S5CQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:42.434321832 (524h17m27.896414305s) [seg_pos={0 0}, upload=461218]: Segment reached optimal threshold of 13 pieces. Oct 15 09:48:42.434605041 (524h17m27.896697534s) [noise=true , address=45.91.116.5:4299 , piece_id=4ARMXXE5LBRE2OZQEIITELKPTW4TISZWTD6XZGI4UJV7UCLUXAEQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX): protocol: expected piece hash; context canceled; EOF Oct 15 09:48:42.434666138 (524h17m27.896758621s) [noise=true , address=45.91.116.5:4299 , piece_id=4ARMXXE5LBRE2OZQEIITELKPTW4TISZWTD6XZGI4UJV7UCLUXAEQ, seg_pos={0 0}, upload=461218]: Long tail cancellation due to optimal threshold Oct 15 09:48:42.434692849 (524h17m27.896785432s) [noise=true , address=45.91.116.12:4226 , piece_id=DRXRGO7OXEKVS5ACL6FC7RKLGBBD6KNI7X6E5DM5D5VV5PEKN7FQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1): protocol: expected piece hash; context canceled; EOF Oct 15 09:48:42.434706184 (524h17m27.896798657s) [noise=true , address=45.91.116.12:4226 , piece_id=DRXRGO7OXEKVS5ACL6FC7RKLGBBD6KNI7X6E5DM5D5VV5PEKN7FQ, seg_pos={0 0}, upload=461218]: Long tail cancellation due to optimal threshold Oct 15 09:48:42.434718318 (524h17m27.896810911s) [noise=true , address=45.91.116.4:4940 , piece_id=3MCSUDYN73NIVJTGKPOVY6N3MALB3GBXI7I5NAX4HXPDHVMQ52BA, seg_pos={0 0}, upload=461218]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe): protocol: expected piece hash; context canceled; EOF Oct 15 09:48:42.434728447 (524h17m27.89682093s) [noise=true , address=45.91.116.4:4940 , piece_id=3MCSUDYN73NIVJTGKPOVY6N3MALB3GBXI7I5NAX4HXPDHVMQ52BA, seg_pos={0 0}, upload=461218]: Long tail cancellation due to optimal threshold Oct 15 09:48:42.434760458 (524h17m27.896852931s) [seg_pos={0 0}, upload=461218]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 15 09:48:42.434866900 (524h17m27.896959384s) [upload=461218]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 15 09:48:42.434873884 (524h17m27.896966367s) [upload=461218]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 15 09:48:42.760633614 (524h17m28.222726087s) [upload=461218]: Done uploading segments... Oct 15 09:48:42.760655435 (524h17m28.222747918s) [upload=461218]: Upload finished. err: <nil> Duration: 6.03582156s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.....................................X...X..................] duration: 6.222781137s FAIL (2025-10-15 10:00:23 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, Piece ID: VH7ZJKSZ2NUXQJZE2IKNWP5PAL3TN2NNHD3ATZUQHGKKYMSSM3LQ) context deadline exceeded; piecestore: (Node ID: 1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N, Piece ID: UMQRQYPZ7SWUCO2N7JDCEVI5U5P36FDL3SUM6HIYJVWVOZTWNXQQ) context deadline exceeded ======LOGS====== Oct 15 09:48:38.801058027 (524h17m24.26315051s) [upload=461218]: Starting upload Oct 15 09:48:38.801138119 (524h17m24.263230592s) [upload=461218]: Done starting upload Oct 15 09:48:38.801288545 (524h17m24.263381028s) [upload=461218]: Uploading segments... Oct 15 09:48:38.801442509 (524h17m24.263535002s) [upload=461218]: Got next segment. Inline: false Oct 15 09:48:38.801486663 (524h17m24.263579136s) [upload=461218]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 15 09:48:38.801493245 (524h17m24.263585728s) [upload=461218]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 15 09:48:39.066131918 (524h17m24.528224401s) [seg_pos={0 0}, upload=461218]: Begin upload segment... Oct 15 09:48:39.066205549 (524h17m24.528298022s) [seg_pos={0 0}, upload=461218]: Done begin upload segment. Oct 15 09:48:39.066243090 (524h17m24.528335573s) [noise=true , address=45.91.116.5:4299 , piece_id=4ARMXXE5LBRE2OZQEIITELKPTW4TISZWTD6XZGI4UJV7UCLUXAEQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.066254431 (524h17m24.528346915s) [noise=true , address=45.91.116.11:4117 , piece_id=VH7ZJKSZ2NUXQJZE2IKNWP5PAL3TN2NNHD3ATZUQHGKKYMSSM3LQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.066874873 (524h17m24.528967356s) [noise=true , address=145.79.227.7:4294 , piece_id=H4ZNVIXORNO4MTXOVYDQZI4IJE4VXHZWC6JI35YKOR3UESSXJNTQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.067301676 (524h17m24.529394159s) [noise=true , address=45.91.116.9:4262 , piece_id=X53JZGRLZLX53UZLLCXXUQCFSQD2FO76X74OM45G26HNDU55UXXQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.067705735 (524h17m24.529798218s) [noise=true , address=145.79.227.4:4187 , piece_id=TQVPL6ZA6LHKAJVL6XWSYASZD3MCKKV3553CRFD72XWTVWY62STQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.069183369 (524h17m24.531275862s) [noise=true , address=145.79.227.5:4222 , piece_id=C2JSPN2QVSGSXG6ZW3ZMOFX5YFNN65XY7E62N5OEHOOMK3NSO4KQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.070534782 (524h17m24.532627265s) [noise=true , address=77.90.31.11:4555 , piece_id=U5RYZSD643KSUI4XLL26EAN6FQRPMG62GQMMCQRPEEG2XKSNBIZQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.072356982 (524h17m24.534449466s) [noise=true , address=145.79.227.11:4730 , piece_id=VO2DSM5LT43QAFPWOY3W3PVEJ44GZJWTCWG5QRQQFB363DFVFZDA, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.073961167 (524h17m24.53605364s) [noise=true , address=45.91.116.4:4940 , piece_id=3MCSUDYN73NIVJTGKPOVY6N3MALB3GBXI7I5NAX4HXPDHVMQ52BA, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.075716960 (524h17m24.537809433s) [noise=true , address=145.79.227.16:4910 , piece_id=BIHEJDH7RPI4UCPUNV453QHSEVCLA54PSNNQB2II37XZDRXFZNWQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.079266419 (524h17m24.541358902s) [noise=true , address=145.79.227.14:4838 , piece_id=6K6A4CNEHGB3X6DEBLF5KHLUT4GE6BQBW6IE3RHBML2NWPN255AA, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.080943032 (524h17m24.543035505s) [noise=true , address=45.91.116.12:4226 , piece_id=DRXRGO7OXEKVS5ACL6FC7RKLGBBD6KNI7X6E5DM5D5VV5PEKN7FQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.081340479 (524h17m24.543432962s) [noise=true , address=45.91.116.8:4045 , piece_id=EDXBNLKKRZMT6MKK2KOIZM7DDKA3IHB6UMCXX4YKZF7JY2T3S5CQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.083616223 (524h17m24.545708706s) [noise=true , address=145.79.227.14:4840 , piece_id=B4ZVPRMVH7Y3TCWCLPO7DWRVMJL7VSI73CIT4PEHCMTIEURQKB2Q, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.083633946 (524h17m24.545726429s) [noise=true , address=45.91.116.14:4189 , piece_id=UMQRQYPZ7SWUCO2N7JDCEVI5U5P36FDL3SUM6HIYJVWVOZTWNXQQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.085493247 (524h17m24.54758573s) [noise=true , address=145.79.227.17:4969 , piece_id=R5HIP33QW5HLICARFWDUOW7I75YTDHZTJCIML5X52VKHLPL3J4YQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:40.727263388 (524h17m26.189355871s) [upload=461218]: Next returned nil segment Oct 15 09:48:40.727310076 (524h17m26.189402549s) [upload=461218]: Waiting for error group managing segments... Oct 15 09:48:41.004514306 (524h17m26.466606789s) [noise=true , address=145.79.227.7:4294 , piece_id=H4ZNVIXORNO4MTXOVYDQZI4IJE4VXHZWC6JI35YKOR3UESSXJNTQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.006116557 (524h17m26.46820904s) [noise=true , address=145.79.227.5:4222 , piece_id=C2JSPN2QVSGSXG6ZW3ZMOFX5YFNN65XY7E62N5OEHOOMK3NSO4KQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.012963184 (524h17m26.475055667s) [noise=true , address=145.79.227.16:4910 , piece_id=BIHEJDH7RPI4UCPUNV453QHSEVCLA54PSNNQB2II37XZDRXFZNWQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.012994684 (524h17m26.475087167s) [noise=true , address=145.79.227.14:4840 , piece_id=B4ZVPRMVH7Y3TCWCLPO7DWRVMJL7VSI73CIT4PEHCMTIEURQKB2Q, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.016649956 (524h17m26.478742439s) [noise=true , address=145.79.227.11:4730 , piece_id=VO2DSM5LT43QAFPWOY3W3PVEJ44GZJWTCWG5QRQQFB363DFVFZDA, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.019248864 (524h17m26.481341347s) [noise=true , address=145.79.227.17:4969 , piece_id=R5HIP33QW5HLICARFWDUOW7I75YTDHZTJCIML5X52VKHLPL3J4YQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.075991585 (524h17m26.538084068s) [noise=true , address=145.79.227.4:4187 , piece_id=TQVPL6ZA6LHKAJVL6XWSYASZD3MCKKV3553CRFD72XWTVWY62STQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.118655146 (524h17m26.580747619s) [noise=true , address=145.79.227.14:4838 , piece_id=6K6A4CNEHGB3X6DEBLF5KHLUT4GE6BQBW6IE3RHBML2NWPN255AA, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.285934795 (524h17m26.748027278s) [noise=true , address=77.90.31.11:4555 , piece_id=U5RYZSD643KSUI4XLL26EAN6FQRPMG62GQMMCQRPEEG2XKSNBIZQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.744525465 (524h17m27.206617948s) [noise=true , address=45.91.116.9:4262 , piece_id=X53JZGRLZLX53UZLLCXXUQCFSQD2FO76X74OM45G26HNDU55UXXQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:42.218961989 (524h17m27.681054462s) [noise=true , address=45.91.116.11:4117 , piece_id=VH7ZJKSZ2NUXQJZE2IKNWP5PAL3TN2NNHD3ATZUQHGKKYMSSM3LQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:42.313638675 (524h17m27.775731158s) [noise=true , address=45.91.116.14:4189 , piece_id=UMQRQYPZ7SWUCO2N7JDCEVI5U5P36FDL3SUM6HIYJVWVOZTWNXQQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:42.434283559 (524h17m27.896376053s) [noise=true , address=45.91.116.8:4045 , piece_id=EDXBNLKKRZMT6MKK2KOIZM7DDKA3IHB6UMCXX4YKZF7JY2T3S5CQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:42.434321832 (524h17m27.896414305s) [seg_pos={0 0}, upload=461218]: Segment reached optimal threshold of 13 pieces. Oct 15 09:48:42.434605041 (524h17m27.896697534s) [noise=true , address=45.91.116.5:4299 , piece_id=4ARMXXE5LBRE2OZQEIITELKPTW4TISZWTD6XZGI4UJV7UCLUXAEQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX): protocol: expected piece hash; context canceled; EOF Oct 15 09:48:42.434666138 (524h17m27.896758621s) [noise=true , address=45.91.116.5:4299 , piece_id=4ARMXXE5LBRE2OZQEIITELKPTW4TISZWTD6XZGI4UJV7UCLUXAEQ, seg_pos={0 0}, upload=461218]: Long tail cancellation due to optimal threshold Oct 15 09:48:42.434692849 (524h17m27.896785432s) [noise=true , address=45.91.116.12:4226 , piece_id=DRXRGO7OXEKVS5ACL6FC7RKLGBBD6KNI7X6E5DM5D5VV5PEKN7FQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1): protocol: expected piece hash; context canceled; EOF Oct 15 09:48:42.434706184 (524h17m27.896798657s) [noise=true , address=45.91.116.12:4226 , piece_id=DRXRGO7OXEKVS5ACL6FC7RKLGBBD6KNI7X6E5DM5D5VV5PEKN7FQ, seg_pos={0 0}, upload=461218]: Long tail cancellation due to optimal threshold Oct 15 09:48:42.434718318 (524h17m27.896810911s) [noise=true , address=45.91.116.4:4940 , piece_id=3MCSUDYN73NIVJTGKPOVY6N3MALB3GBXI7I5NAX4HXPDHVMQ52BA, seg_pos={0 0}, upload=461218]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe): protocol: expected piece hash; context canceled; EOF Oct 15 09:48:42.434728447 (524h17m27.89682093s) [noise=true , address=45.91.116.4:4940 , piece_id=3MCSUDYN73NIVJTGKPOVY6N3MALB3GBXI7I5NAX4HXPDHVMQ52BA, seg_pos={0 0}, upload=461218]: Long tail cancellation due to optimal threshold Oct 15 09:48:42.434760458 (524h17m27.896852931s) [seg_pos={0 0}, upload=461218]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 15 09:48:42.434866900 (524h17m27.896959384s) [upload=461218]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 15 09:48:42.434873884 (524h17m27.896966367s) [upload=461218]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 15 09:48:42.760633614 (524h17m28.222726087s) [upload=461218]: Done uploading segments... Oct 15 09:48:42.760655435 (524h17m28.222747918s) [upload=461218]: Upload finished. err: <nil> Duration: 6.222821233s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [......................................X...X.................] duration: 6.19910795s FAIL (2025-10-15 09:59:17 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, Piece ID: VH7ZJKSZ2NUXQJZE2IKNWP5PAL3TN2NNHD3ATZUQHGKKYMSSM3LQ) context deadline exceeded; piecestore: (Node ID: 1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N, Piece ID: UMQRQYPZ7SWUCO2N7JDCEVI5U5P36FDL3SUM6HIYJVWVOZTWNXQQ) context deadline exceeded ======LOGS====== Oct 15 09:48:38.801058027 (524h17m24.26315051s) [upload=461218]: Starting upload Oct 15 09:48:38.801138119 (524h17m24.263230592s) [upload=461218]: Done starting upload Oct 15 09:48:38.801288545 (524h17m24.263381028s) [upload=461218]: Uploading segments... Oct 15 09:48:38.801442509 (524h17m24.263535002s) [upload=461218]: Got next segment. Inline: false Oct 15 09:48:38.801486663 (524h17m24.263579136s) [upload=461218]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 15 09:48:38.801493245 (524h17m24.263585728s) [upload=461218]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 15 09:48:39.066131918 (524h17m24.528224401s) [seg_pos={0 0}, upload=461218]: Begin upload segment... Oct 15 09:48:39.066205549 (524h17m24.528298022s) [seg_pos={0 0}, upload=461218]: Done begin upload segment. Oct 15 09:48:39.066243090 (524h17m24.528335573s) [noise=true , address=45.91.116.5:4299 , piece_id=4ARMXXE5LBRE2OZQEIITELKPTW4TISZWTD6XZGI4UJV7UCLUXAEQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.066254431 (524h17m24.528346915s) [noise=true , address=45.91.116.11:4117 , piece_id=VH7ZJKSZ2NUXQJZE2IKNWP5PAL3TN2NNHD3ATZUQHGKKYMSSM3LQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.066874873 (524h17m24.528967356s) [noise=true , address=145.79.227.7:4294 , piece_id=H4ZNVIXORNO4MTXOVYDQZI4IJE4VXHZWC6JI35YKOR3UESSXJNTQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.067301676 (524h17m24.529394159s) [noise=true , address=45.91.116.9:4262 , piece_id=X53JZGRLZLX53UZLLCXXUQCFSQD2FO76X74OM45G26HNDU55UXXQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.067705735 (524h17m24.529798218s) [noise=true , address=145.79.227.4:4187 , piece_id=TQVPL6ZA6LHKAJVL6XWSYASZD3MCKKV3553CRFD72XWTVWY62STQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.069183369 (524h17m24.531275862s) [noise=true , address=145.79.227.5:4222 , piece_id=C2JSPN2QVSGSXG6ZW3ZMOFX5YFNN65XY7E62N5OEHOOMK3NSO4KQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.070534782 (524h17m24.532627265s) [noise=true , address=77.90.31.11:4555 , piece_id=U5RYZSD643KSUI4XLL26EAN6FQRPMG62GQMMCQRPEEG2XKSNBIZQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.072356982 (524h17m24.534449466s) [noise=true , address=145.79.227.11:4730 , piece_id=VO2DSM5LT43QAFPWOY3W3PVEJ44GZJWTCWG5QRQQFB363DFVFZDA, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.073961167 (524h17m24.53605364s) [noise=true , address=45.91.116.4:4940 , piece_id=3MCSUDYN73NIVJTGKPOVY6N3MALB3GBXI7I5NAX4HXPDHVMQ52BA, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.075716960 (524h17m24.537809433s) [noise=true , address=145.79.227.16:4910 , piece_id=BIHEJDH7RPI4UCPUNV453QHSEVCLA54PSNNQB2II37XZDRXFZNWQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.079266419 (524h17m24.541358902s) [noise=true , address=145.79.227.14:4838 , piece_id=6K6A4CNEHGB3X6DEBLF5KHLUT4GE6BQBW6IE3RHBML2NWPN255AA, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.080943032 (524h17m24.543035505s) [noise=true , address=45.91.116.12:4226 , piece_id=DRXRGO7OXEKVS5ACL6FC7RKLGBBD6KNI7X6E5DM5D5VV5PEKN7FQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.081340479 (524h17m24.543432962s) [noise=true , address=45.91.116.8:4045 , piece_id=EDXBNLKKRZMT6MKK2KOIZM7DDKA3IHB6UMCXX4YKZF7JY2T3S5CQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.083616223 (524h17m24.545708706s) [noise=true , address=145.79.227.14:4840 , piece_id=B4ZVPRMVH7Y3TCWCLPO7DWRVMJL7VSI73CIT4PEHCMTIEURQKB2Q, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.083633946 (524h17m24.545726429s) [noise=true , address=45.91.116.14:4189 , piece_id=UMQRQYPZ7SWUCO2N7JDCEVI5U5P36FDL3SUM6HIYJVWVOZTWNXQQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.085493247 (524h17m24.54758573s) [noise=true , address=145.79.227.17:4969 , piece_id=R5HIP33QW5HLICARFWDUOW7I75YTDHZTJCIML5X52VKHLPL3J4YQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:40.727263388 (524h17m26.189355871s) [upload=461218]: Next returned nil segment Oct 15 09:48:40.727310076 (524h17m26.189402549s) [upload=461218]: Waiting for error group managing segments... Oct 15 09:48:41.004514306 (524h17m26.466606789s) [noise=true , address=145.79.227.7:4294 , piece_id=H4ZNVIXORNO4MTXOVYDQZI4IJE4VXHZWC6JI35YKOR3UESSXJNTQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.006116557 (524h17m26.46820904s) [noise=true , address=145.79.227.5:4222 , piece_id=C2JSPN2QVSGSXG6ZW3ZMOFX5YFNN65XY7E62N5OEHOOMK3NSO4KQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.012963184 (524h17m26.475055667s) [noise=true , address=145.79.227.16:4910 , piece_id=BIHEJDH7RPI4UCPUNV453QHSEVCLA54PSNNQB2II37XZDRXFZNWQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.012994684 (524h17m26.475087167s) [noise=true , address=145.79.227.14:4840 , piece_id=B4ZVPRMVH7Y3TCWCLPO7DWRVMJL7VSI73CIT4PEHCMTIEURQKB2Q, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.016649956 (524h17m26.478742439s) [noise=true , address=145.79.227.11:4730 , piece_id=VO2DSM5LT43QAFPWOY3W3PVEJ44GZJWTCWG5QRQQFB363DFVFZDA, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.019248864 (524h17m26.481341347s) [noise=true , address=145.79.227.17:4969 , piece_id=R5HIP33QW5HLICARFWDUOW7I75YTDHZTJCIML5X52VKHLPL3J4YQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.075991585 (524h17m26.538084068s) [noise=true , address=145.79.227.4:4187 , piece_id=TQVPL6ZA6LHKAJVL6XWSYASZD3MCKKV3553CRFD72XWTVWY62STQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.118655146 (524h17m26.580747619s) [noise=true , address=145.79.227.14:4838 , piece_id=6K6A4CNEHGB3X6DEBLF5KHLUT4GE6BQBW6IE3RHBML2NWPN255AA, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.285934795 (524h17m26.748027278s) [noise=true , address=77.90.31.11:4555 , piece_id=U5RYZSD643KSUI4XLL26EAN6FQRPMG62GQMMCQRPEEG2XKSNBIZQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.744525465 (524h17m27.206617948s) [noise=true , address=45.91.116.9:4262 , piece_id=X53JZGRLZLX53UZLLCXXUQCFSQD2FO76X74OM45G26HNDU55UXXQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:42.218961989 (524h17m27.681054462s) [noise=true , address=45.91.116.11:4117 , piece_id=VH7ZJKSZ2NUXQJZE2IKNWP5PAL3TN2NNHD3ATZUQHGKKYMSSM3LQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:42.313638675 (524h17m27.775731158s) [noise=true , address=45.91.116.14:4189 , piece_id=UMQRQYPZ7SWUCO2N7JDCEVI5U5P36FDL3SUM6HIYJVWVOZTWNXQQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:42.434283559 (524h17m27.896376053s) [noise=true , address=45.91.116.8:4045 , piece_id=EDXBNLKKRZMT6MKK2KOIZM7DDKA3IHB6UMCXX4YKZF7JY2T3S5CQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:42.434321832 (524h17m27.896414305s) [seg_pos={0 0}, upload=461218]: Segment reached optimal threshold of 13 pieces. Oct 15 09:48:42.434605041 (524h17m27.896697534s) [noise=true , address=45.91.116.5:4299 , piece_id=4ARMXXE5LBRE2OZQEIITELKPTW4TISZWTD6XZGI4UJV7UCLUXAEQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX): protocol: expected piece hash; context canceled; EOF Oct 15 09:48:42.434666138 (524h17m27.896758621s) [noise=true , address=45.91.116.5:4299 , piece_id=4ARMXXE5LBRE2OZQEIITELKPTW4TISZWTD6XZGI4UJV7UCLUXAEQ, seg_pos={0 0}, upload=461218]: Long tail cancellation due to optimal threshold Oct 15 09:48:42.434692849 (524h17m27.896785432s) [noise=true , address=45.91.116.12:4226 , piece_id=DRXRGO7OXEKVS5ACL6FC7RKLGBBD6KNI7X6E5DM5D5VV5PEKN7FQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1): protocol: expected piece hash; context canceled; EOF Oct 15 09:48:42.434706184 (524h17m27.896798657s) [noise=true , address=45.91.116.12:4226 , piece_id=DRXRGO7OXEKVS5ACL6FC7RKLGBBD6KNI7X6E5DM5D5VV5PEKN7FQ, seg_pos={0 0}, upload=461218]: Long tail cancellation due to optimal threshold Oct 15 09:48:42.434718318 (524h17m27.896810911s) [noise=true , address=45.91.116.4:4940 , piece_id=3MCSUDYN73NIVJTGKPOVY6N3MALB3GBXI7I5NAX4HXPDHVMQ52BA, seg_pos={0 0}, upload=461218]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe): protocol: expected piece hash; context canceled; EOF Oct 15 09:48:42.434728447 (524h17m27.89682093s) [noise=true , address=45.91.116.4:4940 , piece_id=3MCSUDYN73NIVJTGKPOVY6N3MALB3GBXI7I5NAX4HXPDHVMQ52BA, seg_pos={0 0}, upload=461218]: Long tail cancellation due to optimal threshold Oct 15 09:48:42.434760458 (524h17m27.896852931s) [seg_pos={0 0}, upload=461218]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 15 09:48:42.434866900 (524h17m27.896959384s) [upload=461218]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 15 09:48:42.434873884 (524h17m27.896966367s) [upload=461218]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 15 09:48:42.760633614 (524h17m28.222726087s) [upload=461218]: Done uploading segments... Oct 15 09:48:42.760655435 (524h17m28.222747918s) [upload=461218]: Upload finished. err: <nil> Duration: 6.199141254s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.......................................X...X................] duration: 6.179595026s FAIL (2025-10-15 09:58:57 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, Piece ID: VH7ZJKSZ2NUXQJZE2IKNWP5PAL3TN2NNHD3ATZUQHGKKYMSSM3LQ) context deadline exceeded; piecestore: (Node ID: 1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N, Piece ID: UMQRQYPZ7SWUCO2N7JDCEVI5U5P36FDL3SUM6HIYJVWVOZTWNXQQ) context deadline exceeded ======LOGS====== Oct 15 09:48:38.801058027 (524h17m24.26315051s) [upload=461218]: Starting upload Oct 15 09:48:38.801138119 (524h17m24.263230592s) [upload=461218]: Done starting upload Oct 15 09:48:38.801288545 (524h17m24.263381028s) [upload=461218]: Uploading segments... Oct 15 09:48:38.801442509 (524h17m24.263535002s) [upload=461218]: Got next segment. Inline: false Oct 15 09:48:38.801486663 (524h17m24.263579136s) [upload=461218]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 15 09:48:38.801493245 (524h17m24.263585728s) [upload=461218]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 15 09:48:39.066131918 (524h17m24.528224401s) [seg_pos={0 0}, upload=461218]: Begin upload segment... Oct 15 09:48:39.066205549 (524h17m24.528298022s) [seg_pos={0 0}, upload=461218]: Done begin upload segment. Oct 15 09:48:39.066243090 (524h17m24.528335573s) [noise=true , address=45.91.116.5:4299 , piece_id=4ARMXXE5LBRE2OZQEIITELKPTW4TISZWTD6XZGI4UJV7UCLUXAEQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.066254431 (524h17m24.528346915s) [noise=true , address=45.91.116.11:4117 , piece_id=VH7ZJKSZ2NUXQJZE2IKNWP5PAL3TN2NNHD3ATZUQHGKKYMSSM3LQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.066874873 (524h17m24.528967356s) [noise=true , address=145.79.227.7:4294 , piece_id=H4ZNVIXORNO4MTXOVYDQZI4IJE4VXHZWC6JI35YKOR3UESSXJNTQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.067301676 (524h17m24.529394159s) [noise=true , address=45.91.116.9:4262 , piece_id=X53JZGRLZLX53UZLLCXXUQCFSQD2FO76X74OM45G26HNDU55UXXQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.067705735 (524h17m24.529798218s) [noise=true , address=145.79.227.4:4187 , piece_id=TQVPL6ZA6LHKAJVL6XWSYASZD3MCKKV3553CRFD72XWTVWY62STQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.069183369 (524h17m24.531275862s) [noise=true , address=145.79.227.5:4222 , piece_id=C2JSPN2QVSGSXG6ZW3ZMOFX5YFNN65XY7E62N5OEHOOMK3NSO4KQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.070534782 (524h17m24.532627265s) [noise=true , address=77.90.31.11:4555 , piece_id=U5RYZSD643KSUI4XLL26EAN6FQRPMG62GQMMCQRPEEG2XKSNBIZQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.072356982 (524h17m24.534449466s) [noise=true , address=145.79.227.11:4730 , piece_id=VO2DSM5LT43QAFPWOY3W3PVEJ44GZJWTCWG5QRQQFB363DFVFZDA, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.073961167 (524h17m24.53605364s) [noise=true , address=45.91.116.4:4940 , piece_id=3MCSUDYN73NIVJTGKPOVY6N3MALB3GBXI7I5NAX4HXPDHVMQ52BA, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.075716960 (524h17m24.537809433s) [noise=true , address=145.79.227.16:4910 , piece_id=BIHEJDH7RPI4UCPUNV453QHSEVCLA54PSNNQB2II37XZDRXFZNWQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.079266419 (524h17m24.541358902s) [noise=true , address=145.79.227.14:4838 , piece_id=6K6A4CNEHGB3X6DEBLF5KHLUT4GE6BQBW6IE3RHBML2NWPN255AA, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.080943032 (524h17m24.543035505s) [noise=true , address=45.91.116.12:4226 , piece_id=DRXRGO7OXEKVS5ACL6FC7RKLGBBD6KNI7X6E5DM5D5VV5PEKN7FQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.081340479 (524h17m24.543432962s) [noise=true , address=45.91.116.8:4045 , piece_id=EDXBNLKKRZMT6MKK2KOIZM7DDKA3IHB6UMCXX4YKZF7JY2T3S5CQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.083616223 (524h17m24.545708706s) [noise=true , address=145.79.227.14:4840 , piece_id=B4ZVPRMVH7Y3TCWCLPO7DWRVMJL7VSI73CIT4PEHCMTIEURQKB2Q, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.083633946 (524h17m24.545726429s) [noise=true , address=45.91.116.14:4189 , piece_id=UMQRQYPZ7SWUCO2N7JDCEVI5U5P36FDL3SUM6HIYJVWVOZTWNXQQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.085493247 (524h17m24.54758573s) [noise=true , address=145.79.227.17:4969 , piece_id=R5HIP33QW5HLICARFWDUOW7I75YTDHZTJCIML5X52VKHLPL3J4YQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:40.727263388 (524h17m26.189355871s) [upload=461218]: Next returned nil segment Oct 15 09:48:40.727310076 (524h17m26.189402549s) [upload=461218]: Waiting for error group managing segments... Oct 15 09:48:41.004514306 (524h17m26.466606789s) [noise=true , address=145.79.227.7:4294 , piece_id=H4ZNVIXORNO4MTXOVYDQZI4IJE4VXHZWC6JI35YKOR3UESSXJNTQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.006116557 (524h17m26.46820904s) [noise=true , address=145.79.227.5:4222 , piece_id=C2JSPN2QVSGSXG6ZW3ZMOFX5YFNN65XY7E62N5OEHOOMK3NSO4KQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.012963184 (524h17m26.475055667s) [noise=true , address=145.79.227.16:4910 , piece_id=BIHEJDH7RPI4UCPUNV453QHSEVCLA54PSNNQB2II37XZDRXFZNWQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.012994684 (524h17m26.475087167s) [noise=true , address=145.79.227.14:4840 , piece_id=B4ZVPRMVH7Y3TCWCLPO7DWRVMJL7VSI73CIT4PEHCMTIEURQKB2Q, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.016649956 (524h17m26.478742439s) [noise=true , address=145.79.227.11:4730 , piece_id=VO2DSM5LT43QAFPWOY3W3PVEJ44GZJWTCWG5QRQQFB363DFVFZDA, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.019248864 (524h17m26.481341347s) [noise=true , address=145.79.227.17:4969 , piece_id=R5HIP33QW5HLICARFWDUOW7I75YTDHZTJCIML5X52VKHLPL3J4YQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.075991585 (524h17m26.538084068s) [noise=true , address=145.79.227.4:4187 , piece_id=TQVPL6ZA6LHKAJVL6XWSYASZD3MCKKV3553CRFD72XWTVWY62STQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.118655146 (524h17m26.580747619s) [noise=true , address=145.79.227.14:4838 , piece_id=6K6A4CNEHGB3X6DEBLF5KHLUT4GE6BQBW6IE3RHBML2NWPN255AA, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.285934795 (524h17m26.748027278s) [noise=true , address=77.90.31.11:4555 , piece_id=U5RYZSD643KSUI4XLL26EAN6FQRPMG62GQMMCQRPEEG2XKSNBIZQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.744525465 (524h17m27.206617948s) [noise=true , address=45.91.116.9:4262 , piece_id=X53JZGRLZLX53UZLLCXXUQCFSQD2FO76X74OM45G26HNDU55UXXQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:42.218961989 (524h17m27.681054462s) [noise=true , address=45.91.116.11:4117 , piece_id=VH7ZJKSZ2NUXQJZE2IKNWP5PAL3TN2NNHD3ATZUQHGKKYMSSM3LQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:42.313638675 (524h17m27.775731158s) [noise=true , address=45.91.116.14:4189 , piece_id=UMQRQYPZ7SWUCO2N7JDCEVI5U5P36FDL3SUM6HIYJVWVOZTWNXQQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:42.434283559 (524h17m27.896376053s) [noise=true , address=45.91.116.8:4045 , piece_id=EDXBNLKKRZMT6MKK2KOIZM7DDKA3IHB6UMCXX4YKZF7JY2T3S5CQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:42.434321832 (524h17m27.896414305s) [seg_pos={0 0}, upload=461218]: Segment reached optimal threshold of 13 pieces. Oct 15 09:48:42.434605041 (524h17m27.896697534s) [noise=true , address=45.91.116.5:4299 , piece_id=4ARMXXE5LBRE2OZQEIITELKPTW4TISZWTD6XZGI4UJV7UCLUXAEQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX): protocol: expected piece hash; context canceled; EOF Oct 15 09:48:42.434666138 (524h17m27.896758621s) [noise=true , address=45.91.116.5:4299 , piece_id=4ARMXXE5LBRE2OZQEIITELKPTW4TISZWTD6XZGI4UJV7UCLUXAEQ, seg_pos={0 0}, upload=461218]: Long tail cancellation due to optimal threshold Oct 15 09:48:42.434692849 (524h17m27.896785432s) [noise=true , address=45.91.116.12:4226 , piece_id=DRXRGO7OXEKVS5ACL6FC7RKLGBBD6KNI7X6E5DM5D5VV5PEKN7FQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1): protocol: expected piece hash; context canceled; EOF Oct 15 09:48:42.434706184 (524h17m27.896798657s) [noise=true , address=45.91.116.12:4226 , piece_id=DRXRGO7OXEKVS5ACL6FC7RKLGBBD6KNI7X6E5DM5D5VV5PEKN7FQ, seg_pos={0 0}, upload=461218]: Long tail cancellation due to optimal threshold Oct 15 09:48:42.434718318 (524h17m27.896810911s) [noise=true , address=45.91.116.4:4940 , piece_id=3MCSUDYN73NIVJTGKPOVY6N3MALB3GBXI7I5NAX4HXPDHVMQ52BA, seg_pos={0 0}, upload=461218]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe): protocol: expected piece hash; context canceled; EOF Oct 15 09:48:42.434728447 (524h17m27.89682093s) [noise=true , address=45.91.116.4:4940 , piece_id=3MCSUDYN73NIVJTGKPOVY6N3MALB3GBXI7I5NAX4HXPDHVMQ52BA, seg_pos={0 0}, upload=461218]: Long tail cancellation due to optimal threshold Oct 15 09:48:42.434760458 (524h17m27.896852931s) [seg_pos={0 0}, upload=461218]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 15 09:48:42.434866900 (524h17m27.896959384s) [upload=461218]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 15 09:48:42.434873884 (524h17m27.896966367s) [upload=461218]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 15 09:48:42.760633614 (524h17m28.222726087s) [upload=461218]: Done uploading segments... Oct 15 09:48:42.760655435 (524h17m28.222747918s) [upload=461218]: Upload finished. err: <nil> Duration: 6.179624994s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [........................................X...X...............] duration: 6.012853163s FAIL (2025-10-15 09:58:26 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, Piece ID: VH7ZJKSZ2NUXQJZE2IKNWP5PAL3TN2NNHD3ATZUQHGKKYMSSM3LQ) context deadline exceeded; piecestore: (Node ID: 1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N, Piece ID: UMQRQYPZ7SWUCO2N7JDCEVI5U5P36FDL3SUM6HIYJVWVOZTWNXQQ) context deadline exceeded ======LOGS====== Oct 15 09:48:38.801058027 (524h17m24.26315051s) [upload=461218]: Starting upload Oct 15 09:48:38.801138119 (524h17m24.263230592s) [upload=461218]: Done starting upload Oct 15 09:48:38.801288545 (524h17m24.263381028s) [upload=461218]: Uploading segments... Oct 15 09:48:38.801442509 (524h17m24.263535002s) [upload=461218]: Got next segment. Inline: false Oct 15 09:48:38.801486663 (524h17m24.263579136s) [upload=461218]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 15 09:48:38.801493245 (524h17m24.263585728s) [upload=461218]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 15 09:48:39.066131918 (524h17m24.528224401s) [seg_pos={0 0}, upload=461218]: Begin upload segment... Oct 15 09:48:39.066205549 (524h17m24.528298022s) [seg_pos={0 0}, upload=461218]: Done begin upload segment. Oct 15 09:48:39.066243090 (524h17m24.528335573s) [noise=true , address=45.91.116.5:4299 , piece_id=4ARMXXE5LBRE2OZQEIITELKPTW4TISZWTD6XZGI4UJV7UCLUXAEQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.066254431 (524h17m24.528346915s) [noise=true , address=45.91.116.11:4117 , piece_id=VH7ZJKSZ2NUXQJZE2IKNWP5PAL3TN2NNHD3ATZUQHGKKYMSSM3LQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.066874873 (524h17m24.528967356s) [noise=true , address=145.79.227.7:4294 , piece_id=H4ZNVIXORNO4MTXOVYDQZI4IJE4VXHZWC6JI35YKOR3UESSXJNTQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.067301676 (524h17m24.529394159s) [noise=true , address=45.91.116.9:4262 , piece_id=X53JZGRLZLX53UZLLCXXUQCFSQD2FO76X74OM45G26HNDU55UXXQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.067705735 (524h17m24.529798218s) [noise=true , address=145.79.227.4:4187 , piece_id=TQVPL6ZA6LHKAJVL6XWSYASZD3MCKKV3553CRFD72XWTVWY62STQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.069183369 (524h17m24.531275862s) [noise=true , address=145.79.227.5:4222 , piece_id=C2JSPN2QVSGSXG6ZW3ZMOFX5YFNN65XY7E62N5OEHOOMK3NSO4KQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.070534782 (524h17m24.532627265s) [noise=true , address=77.90.31.11:4555 , piece_id=U5RYZSD643KSUI4XLL26EAN6FQRPMG62GQMMCQRPEEG2XKSNBIZQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.072356982 (524h17m24.534449466s) [noise=true , address=145.79.227.11:4730 , piece_id=VO2DSM5LT43QAFPWOY3W3PVEJ44GZJWTCWG5QRQQFB363DFVFZDA, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.073961167 (524h17m24.53605364s) [noise=true , address=45.91.116.4:4940 , piece_id=3MCSUDYN73NIVJTGKPOVY6N3MALB3GBXI7I5NAX4HXPDHVMQ52BA, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.075716960 (524h17m24.537809433s) [noise=true , address=145.79.227.16:4910 , piece_id=BIHEJDH7RPI4UCPUNV453QHSEVCLA54PSNNQB2II37XZDRXFZNWQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.079266419 (524h17m24.541358902s) [noise=true , address=145.79.227.14:4838 , piece_id=6K6A4CNEHGB3X6DEBLF5KHLUT4GE6BQBW6IE3RHBML2NWPN255AA, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.080943032 (524h17m24.543035505s) [noise=true , address=45.91.116.12:4226 , piece_id=DRXRGO7OXEKVS5ACL6FC7RKLGBBD6KNI7X6E5DM5D5VV5PEKN7FQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.081340479 (524h17m24.543432962s) [noise=true , address=45.91.116.8:4045 , piece_id=EDXBNLKKRZMT6MKK2KOIZM7DDKA3IHB6UMCXX4YKZF7JY2T3S5CQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.083616223 (524h17m24.545708706s) [noise=true , address=145.79.227.14:4840 , piece_id=B4ZVPRMVH7Y3TCWCLPO7DWRVMJL7VSI73CIT4PEHCMTIEURQKB2Q, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.083633946 (524h17m24.545726429s) [noise=true , address=45.91.116.14:4189 , piece_id=UMQRQYPZ7SWUCO2N7JDCEVI5U5P36FDL3SUM6HIYJVWVOZTWNXQQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.085493247 (524h17m24.54758573s) [noise=true , address=145.79.227.17:4969 , piece_id=R5HIP33QW5HLICARFWDUOW7I75YTDHZTJCIML5X52VKHLPL3J4YQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:40.727263388 (524h17m26.189355871s) [upload=461218]: Next returned nil segment Oct 15 09:48:40.727310076 (524h17m26.189402549s) [upload=461218]: Waiting for error group managing segments... Oct 15 09:48:41.004514306 (524h17m26.466606789s) [noise=true , address=145.79.227.7:4294 , piece_id=H4ZNVIXORNO4MTXOVYDQZI4IJE4VXHZWC6JI35YKOR3UESSXJNTQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.006116557 (524h17m26.46820904s) [noise=true , address=145.79.227.5:4222 , piece_id=C2JSPN2QVSGSXG6ZW3ZMOFX5YFNN65XY7E62N5OEHOOMK3NSO4KQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.012963184 (524h17m26.475055667s) [noise=true , address=145.79.227.16:4910 , piece_id=BIHEJDH7RPI4UCPUNV453QHSEVCLA54PSNNQB2II37XZDRXFZNWQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.012994684 (524h17m26.475087167s) [noise=true , address=145.79.227.14:4840 , piece_id=B4ZVPRMVH7Y3TCWCLPO7DWRVMJL7VSI73CIT4PEHCMTIEURQKB2Q, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.016649956 (524h17m26.478742439s) [noise=true , address=145.79.227.11:4730 , piece_id=VO2DSM5LT43QAFPWOY3W3PVEJ44GZJWTCWG5QRQQFB363DFVFZDA, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.019248864 (524h17m26.481341347s) [noise=true , address=145.79.227.17:4969 , piece_id=R5HIP33QW5HLICARFWDUOW7I75YTDHZTJCIML5X52VKHLPL3J4YQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.075991585 (524h17m26.538084068s) [noise=true , address=145.79.227.4:4187 , piece_id=TQVPL6ZA6LHKAJVL6XWSYASZD3MCKKV3553CRFD72XWTVWY62STQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.118655146 (524h17m26.580747619s) [noise=true , address=145.79.227.14:4838 , piece_id=6K6A4CNEHGB3X6DEBLF5KHLUT4GE6BQBW6IE3RHBML2NWPN255AA, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.285934795 (524h17m26.748027278s) [noise=true , address=77.90.31.11:4555 , piece_id=U5RYZSD643KSUI4XLL26EAN6FQRPMG62GQMMCQRPEEG2XKSNBIZQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.744525465 (524h17m27.206617948s) [noise=true , address=45.91.116.9:4262 , piece_id=X53JZGRLZLX53UZLLCXXUQCFSQD2FO76X74OM45G26HNDU55UXXQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:42.218961989 (524h17m27.681054462s) [noise=true , address=45.91.116.11:4117 , piece_id=VH7ZJKSZ2NUXQJZE2IKNWP5PAL3TN2NNHD3ATZUQHGKKYMSSM3LQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:42.313638675 (524h17m27.775731158s) [noise=true , address=45.91.116.14:4189 , piece_id=UMQRQYPZ7SWUCO2N7JDCEVI5U5P36FDL3SUM6HIYJVWVOZTWNXQQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:42.434283559 (524h17m27.896376053s) [noise=true , address=45.91.116.8:4045 , piece_id=EDXBNLKKRZMT6MKK2KOIZM7DDKA3IHB6UMCXX4YKZF7JY2T3S5CQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:42.434321832 (524h17m27.896414305s) [seg_pos={0 0}, upload=461218]: Segment reached optimal threshold of 13 pieces. Oct 15 09:48:42.434605041 (524h17m27.896697534s) [noise=true , address=45.91.116.5:4299 , piece_id=4ARMXXE5LBRE2OZQEIITELKPTW4TISZWTD6XZGI4UJV7UCLUXAEQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX): protocol: expected piece hash; context canceled; EOF Oct 15 09:48:42.434666138 (524h17m27.896758621s) [noise=true , address=45.91.116.5:4299 , piece_id=4ARMXXE5LBRE2OZQEIITELKPTW4TISZWTD6XZGI4UJV7UCLUXAEQ, seg_pos={0 0}, upload=461218]: Long tail cancellation due to optimal threshold Oct 15 09:48:42.434692849 (524h17m27.896785432s) [noise=true , address=45.91.116.12:4226 , piece_id=DRXRGO7OXEKVS5ACL6FC7RKLGBBD6KNI7X6E5DM5D5VV5PEKN7FQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1): protocol: expected piece hash; context canceled; EOF Oct 15 09:48:42.434706184 (524h17m27.896798657s) [noise=true , address=45.91.116.12:4226 , piece_id=DRXRGO7OXEKVS5ACL6FC7RKLGBBD6KNI7X6E5DM5D5VV5PEKN7FQ, seg_pos={0 0}, upload=461218]: Long tail cancellation due to optimal threshold Oct 15 09:48:42.434718318 (524h17m27.896810911s) [noise=true , address=45.91.116.4:4940 , piece_id=3MCSUDYN73NIVJTGKPOVY6N3MALB3GBXI7I5NAX4HXPDHVMQ52BA, seg_pos={0 0}, upload=461218]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe): protocol: expected piece hash; context canceled; EOF Oct 15 09:48:42.434728447 (524h17m27.89682093s) [noise=true , address=45.91.116.4:4940 , piece_id=3MCSUDYN73NIVJTGKPOVY6N3MALB3GBXI7I5NAX4HXPDHVMQ52BA, seg_pos={0 0}, upload=461218]: Long tail cancellation due to optimal threshold Oct 15 09:48:42.434760458 (524h17m27.896852931s) [seg_pos={0 0}, upload=461218]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 15 09:48:42.434866900 (524h17m27.896959384s) [upload=461218]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 15 09:48:42.434873884 (524h17m27.896966367s) [upload=461218]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 15 09:48:42.760633614 (524h17m28.222726087s) [upload=461218]: Done uploading segments... Oct 15 09:48:42.760655435 (524h17m28.222747918s) [upload=461218]: Upload finished. err: <nil> Duration: 6.012886376s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.........................................X...X..............] duration: 5.918458464s FAIL (2025-10-15 09:57:51 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, Piece ID: VH7ZJKSZ2NUXQJZE2IKNWP5PAL3TN2NNHD3ATZUQHGKKYMSSM3LQ) context deadline exceeded; piecestore: (Node ID: 1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N, Piece ID: UMQRQYPZ7SWUCO2N7JDCEVI5U5P36FDL3SUM6HIYJVWVOZTWNXQQ) context deadline exceeded ======LOGS====== Oct 15 09:48:38.801058027 (524h17m24.26315051s) [upload=461218]: Starting upload Oct 15 09:48:38.801138119 (524h17m24.263230592s) [upload=461218]: Done starting upload Oct 15 09:48:38.801288545 (524h17m24.263381028s) [upload=461218]: Uploading segments... Oct 15 09:48:38.801442509 (524h17m24.263535002s) [upload=461218]: Got next segment. Inline: false Oct 15 09:48:38.801486663 (524h17m24.263579136s) [upload=461218]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 15 09:48:38.801493245 (524h17m24.263585728s) [upload=461218]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 15 09:48:39.066131918 (524h17m24.528224401s) [seg_pos={0 0}, upload=461218]: Begin upload segment... Oct 15 09:48:39.066205549 (524h17m24.528298022s) [seg_pos={0 0}, upload=461218]: Done begin upload segment. Oct 15 09:48:39.066243090 (524h17m24.528335573s) [noise=true , address=45.91.116.5:4299 , piece_id=4ARMXXE5LBRE2OZQEIITELKPTW4TISZWTD6XZGI4UJV7UCLUXAEQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.066254431 (524h17m24.528346915s) [noise=true , address=45.91.116.11:4117 , piece_id=VH7ZJKSZ2NUXQJZE2IKNWP5PAL3TN2NNHD3ATZUQHGKKYMSSM3LQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.066874873 (524h17m24.528967356s) [noise=true , address=145.79.227.7:4294 , piece_id=H4ZNVIXORNO4MTXOVYDQZI4IJE4VXHZWC6JI35YKOR3UESSXJNTQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.067301676 (524h17m24.529394159s) [noise=true , address=45.91.116.9:4262 , piece_id=X53JZGRLZLX53UZLLCXXUQCFSQD2FO76X74OM45G26HNDU55UXXQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.067705735 (524h17m24.529798218s) [noise=true , address=145.79.227.4:4187 , piece_id=TQVPL6ZA6LHKAJVL6XWSYASZD3MCKKV3553CRFD72XWTVWY62STQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.069183369 (524h17m24.531275862s) [noise=true , address=145.79.227.5:4222 , piece_id=C2JSPN2QVSGSXG6ZW3ZMOFX5YFNN65XY7E62N5OEHOOMK3NSO4KQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.070534782 (524h17m24.532627265s) [noise=true , address=77.90.31.11:4555 , piece_id=U5RYZSD643KSUI4XLL26EAN6FQRPMG62GQMMCQRPEEG2XKSNBIZQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.072356982 (524h17m24.534449466s) [noise=true , address=145.79.227.11:4730 , piece_id=VO2DSM5LT43QAFPWOY3W3PVEJ44GZJWTCWG5QRQQFB363DFVFZDA, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.073961167 (524h17m24.53605364s) [noise=true , address=45.91.116.4:4940 , piece_id=3MCSUDYN73NIVJTGKPOVY6N3MALB3GBXI7I5NAX4HXPDHVMQ52BA, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.075716960 (524h17m24.537809433s) [noise=true , address=145.79.227.16:4910 , piece_id=BIHEJDH7RPI4UCPUNV453QHSEVCLA54PSNNQB2II37XZDRXFZNWQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.079266419 (524h17m24.541358902s) [noise=true , address=145.79.227.14:4838 , piece_id=6K6A4CNEHGB3X6DEBLF5KHLUT4GE6BQBW6IE3RHBML2NWPN255AA, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.080943032 (524h17m24.543035505s) [noise=true , address=45.91.116.12:4226 , piece_id=DRXRGO7OXEKVS5ACL6FC7RKLGBBD6KNI7X6E5DM5D5VV5PEKN7FQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.081340479 (524h17m24.543432962s) [noise=true , address=45.91.116.8:4045 , piece_id=EDXBNLKKRZMT6MKK2KOIZM7DDKA3IHB6UMCXX4YKZF7JY2T3S5CQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.083616223 (524h17m24.545708706s) [noise=true , address=145.79.227.14:4840 , piece_id=B4ZVPRMVH7Y3TCWCLPO7DWRVMJL7VSI73CIT4PEHCMTIEURQKB2Q, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.083633946 (524h17m24.545726429s) [noise=true , address=45.91.116.14:4189 , piece_id=UMQRQYPZ7SWUCO2N7JDCEVI5U5P36FDL3SUM6HIYJVWVOZTWNXQQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.085493247 (524h17m24.54758573s) [noise=true , address=145.79.227.17:4969 , piece_id=R5HIP33QW5HLICARFWDUOW7I75YTDHZTJCIML5X52VKHLPL3J4YQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:40.727263388 (524h17m26.189355871s) [upload=461218]: Next returned nil segment Oct 15 09:48:40.727310076 (524h17m26.189402549s) [upload=461218]: Waiting for error group managing segments... Oct 15 09:48:41.004514306 (524h17m26.466606789s) [noise=true , address=145.79.227.7:4294 , piece_id=H4ZNVIXORNO4MTXOVYDQZI4IJE4VXHZWC6JI35YKOR3UESSXJNTQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.006116557 (524h17m26.46820904s) [noise=true , address=145.79.227.5:4222 , piece_id=C2JSPN2QVSGSXG6ZW3ZMOFX5YFNN65XY7E62N5OEHOOMK3NSO4KQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.012963184 (524h17m26.475055667s) [noise=true , address=145.79.227.16:4910 , piece_id=BIHEJDH7RPI4UCPUNV453QHSEVCLA54PSNNQB2II37XZDRXFZNWQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.012994684 (524h17m26.475087167s) [noise=true , address=145.79.227.14:4840 , piece_id=B4ZVPRMVH7Y3TCWCLPO7DWRVMJL7VSI73CIT4PEHCMTIEURQKB2Q, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.016649956 (524h17m26.478742439s) [noise=true , address=145.79.227.11:4730 , piece_id=VO2DSM5LT43QAFPWOY3W3PVEJ44GZJWTCWG5QRQQFB363DFVFZDA, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.019248864 (524h17m26.481341347s) [noise=true , address=145.79.227.17:4969 , piece_id=R5HIP33QW5HLICARFWDUOW7I75YTDHZTJCIML5X52VKHLPL3J4YQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.075991585 (524h17m26.538084068s) [noise=true , address=145.79.227.4:4187 , piece_id=TQVPL6ZA6LHKAJVL6XWSYASZD3MCKKV3553CRFD72XWTVWY62STQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.118655146 (524h17m26.580747619s) [noise=true , address=145.79.227.14:4838 , piece_id=6K6A4CNEHGB3X6DEBLF5KHLUT4GE6BQBW6IE3RHBML2NWPN255AA, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.285934795 (524h17m26.748027278s) [noise=true , address=77.90.31.11:4555 , piece_id=U5RYZSD643KSUI4XLL26EAN6FQRPMG62GQMMCQRPEEG2XKSNBIZQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.744525465 (524h17m27.206617948s) [noise=true , address=45.91.116.9:4262 , piece_id=X53JZGRLZLX53UZLLCXXUQCFSQD2FO76X74OM45G26HNDU55UXXQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:42.218961989 (524h17m27.681054462s) [noise=true , address=45.91.116.11:4117 , piece_id=VH7ZJKSZ2NUXQJZE2IKNWP5PAL3TN2NNHD3ATZUQHGKKYMSSM3LQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:42.313638675 (524h17m27.775731158s) [noise=true , address=45.91.116.14:4189 , piece_id=UMQRQYPZ7SWUCO2N7JDCEVI5U5P36FDL3SUM6HIYJVWVOZTWNXQQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:42.434283559 (524h17m27.896376053s) [noise=true , address=45.91.116.8:4045 , piece_id=EDXBNLKKRZMT6MKK2KOIZM7DDKA3IHB6UMCXX4YKZF7JY2T3S5CQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:42.434321832 (524h17m27.896414305s) [seg_pos={0 0}, upload=461218]: Segment reached optimal threshold of 13 pieces. Oct 15 09:48:42.434605041 (524h17m27.896697534s) [noise=true , address=45.91.116.5:4299 , piece_id=4ARMXXE5LBRE2OZQEIITELKPTW4TISZWTD6XZGI4UJV7UCLUXAEQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX): protocol: expected piece hash; context canceled; EOF Oct 15 09:48:42.434666138 (524h17m27.896758621s) [noise=true , address=45.91.116.5:4299 , piece_id=4ARMXXE5LBRE2OZQEIITELKPTW4TISZWTD6XZGI4UJV7UCLUXAEQ, seg_pos={0 0}, upload=461218]: Long tail cancellation due to optimal threshold Oct 15 09:48:42.434692849 (524h17m27.896785432s) [noise=true , address=45.91.116.12:4226 , piece_id=DRXRGO7OXEKVS5ACL6FC7RKLGBBD6KNI7X6E5DM5D5VV5PEKN7FQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1): protocol: expected piece hash; context canceled; EOF Oct 15 09:48:42.434706184 (524h17m27.896798657s) [noise=true , address=45.91.116.12:4226 , piece_id=DRXRGO7OXEKVS5ACL6FC7RKLGBBD6KNI7X6E5DM5D5VV5PEKN7FQ, seg_pos={0 0}, upload=461218]: Long tail cancellation due to optimal threshold Oct 15 09:48:42.434718318 (524h17m27.896810911s) [noise=true , address=45.91.116.4:4940 , piece_id=3MCSUDYN73NIVJTGKPOVY6N3MALB3GBXI7I5NAX4HXPDHVMQ52BA, seg_pos={0 0}, upload=461218]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe): protocol: expected piece hash; context canceled; EOF Oct 15 09:48:42.434728447 (524h17m27.89682093s) [noise=true , address=45.91.116.4:4940 , piece_id=3MCSUDYN73NIVJTGKPOVY6N3MALB3GBXI7I5NAX4HXPDHVMQ52BA, seg_pos={0 0}, upload=461218]: Long tail cancellation due to optimal threshold Oct 15 09:48:42.434760458 (524h17m27.896852931s) [seg_pos={0 0}, upload=461218]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 15 09:48:42.434866900 (524h17m27.896959384s) [upload=461218]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 15 09:48:42.434873884 (524h17m27.896966367s) [upload=461218]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 15 09:48:42.760633614 (524h17m28.222726087s) [upload=461218]: Done uploading segments... Oct 15 09:48:42.760655435 (524h17m28.222747918s) [upload=461218]: Upload finished. err: <nil> Duration: 5.918507186s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..........................................X...X.............] duration: 6.031964464s FAIL (2025-10-15 09:57:26 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, Piece ID: VH7ZJKSZ2NUXQJZE2IKNWP5PAL3TN2NNHD3ATZUQHGKKYMSSM3LQ) context deadline exceeded; piecestore: (Node ID: 1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N, Piece ID: UMQRQYPZ7SWUCO2N7JDCEVI5U5P36FDL3SUM6HIYJVWVOZTWNXQQ) context deadline exceeded ======LOGS====== Oct 15 09:48:38.801058027 (524h17m24.26315051s) [upload=461218]: Starting upload Oct 15 09:48:38.801138119 (524h17m24.263230592s) [upload=461218]: Done starting upload Oct 15 09:48:38.801288545 (524h17m24.263381028s) [upload=461218]: Uploading segments... Oct 15 09:48:38.801442509 (524h17m24.263535002s) [upload=461218]: Got next segment. Inline: false Oct 15 09:48:38.801486663 (524h17m24.263579136s) [upload=461218]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 15 09:48:38.801493245 (524h17m24.263585728s) [upload=461218]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 15 09:48:39.066131918 (524h17m24.528224401s) [seg_pos={0 0}, upload=461218]: Begin upload segment... Oct 15 09:48:39.066205549 (524h17m24.528298022s) [seg_pos={0 0}, upload=461218]: Done begin upload segment. Oct 15 09:48:39.066243090 (524h17m24.528335573s) [noise=true , address=45.91.116.5:4299 , piece_id=4ARMXXE5LBRE2OZQEIITELKPTW4TISZWTD6XZGI4UJV7UCLUXAEQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.066254431 (524h17m24.528346915s) [noise=true , address=45.91.116.11:4117 , piece_id=VH7ZJKSZ2NUXQJZE2IKNWP5PAL3TN2NNHD3ATZUQHGKKYMSSM3LQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.066874873 (524h17m24.528967356s) [noise=true , address=145.79.227.7:4294 , piece_id=H4ZNVIXORNO4MTXOVYDQZI4IJE4VXHZWC6JI35YKOR3UESSXJNTQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.067301676 (524h17m24.529394159s) [noise=true , address=45.91.116.9:4262 , piece_id=X53JZGRLZLX53UZLLCXXUQCFSQD2FO76X74OM45G26HNDU55UXXQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.067705735 (524h17m24.529798218s) [noise=true , address=145.79.227.4:4187 , piece_id=TQVPL6ZA6LHKAJVL6XWSYASZD3MCKKV3553CRFD72XWTVWY62STQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.069183369 (524h17m24.531275862s) [noise=true , address=145.79.227.5:4222 , piece_id=C2JSPN2QVSGSXG6ZW3ZMOFX5YFNN65XY7E62N5OEHOOMK3NSO4KQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.070534782 (524h17m24.532627265s) [noise=true , address=77.90.31.11:4555 , piece_id=U5RYZSD643KSUI4XLL26EAN6FQRPMG62GQMMCQRPEEG2XKSNBIZQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.072356982 (524h17m24.534449466s) [noise=true , address=145.79.227.11:4730 , piece_id=VO2DSM5LT43QAFPWOY3W3PVEJ44GZJWTCWG5QRQQFB363DFVFZDA, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.073961167 (524h17m24.53605364s) [noise=true , address=45.91.116.4:4940 , piece_id=3MCSUDYN73NIVJTGKPOVY6N3MALB3GBXI7I5NAX4HXPDHVMQ52BA, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.075716960 (524h17m24.537809433s) [noise=true , address=145.79.227.16:4910 , piece_id=BIHEJDH7RPI4UCPUNV453QHSEVCLA54PSNNQB2II37XZDRXFZNWQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.079266419 (524h17m24.541358902s) [noise=true , address=145.79.227.14:4838 , piece_id=6K6A4CNEHGB3X6DEBLF5KHLUT4GE6BQBW6IE3RHBML2NWPN255AA, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.080943032 (524h17m24.543035505s) [noise=true , address=45.91.116.12:4226 , piece_id=DRXRGO7OXEKVS5ACL6FC7RKLGBBD6KNI7X6E5DM5D5VV5PEKN7FQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.081340479 (524h17m24.543432962s) [noise=true , address=45.91.116.8:4045 , piece_id=EDXBNLKKRZMT6MKK2KOIZM7DDKA3IHB6UMCXX4YKZF7JY2T3S5CQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.083616223 (524h17m24.545708706s) [noise=true , address=145.79.227.14:4840 , piece_id=B4ZVPRMVH7Y3TCWCLPO7DWRVMJL7VSI73CIT4PEHCMTIEURQKB2Q, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.083633946 (524h17m24.545726429s) [noise=true , address=45.91.116.14:4189 , piece_id=UMQRQYPZ7SWUCO2N7JDCEVI5U5P36FDL3SUM6HIYJVWVOZTWNXQQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.085493247 (524h17m24.54758573s) [noise=true , address=145.79.227.17:4969 , piece_id=R5HIP33QW5HLICARFWDUOW7I75YTDHZTJCIML5X52VKHLPL3J4YQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:40.727263388 (524h17m26.189355871s) [upload=461218]: Next returned nil segment Oct 15 09:48:40.727310076 (524h17m26.189402549s) [upload=461218]: Waiting for error group managing segments... Oct 15 09:48:41.004514306 (524h17m26.466606789s) [noise=true , address=145.79.227.7:4294 , piece_id=H4ZNVIXORNO4MTXOVYDQZI4IJE4VXHZWC6JI35YKOR3UESSXJNTQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.006116557 (524h17m26.46820904s) [noise=true , address=145.79.227.5:4222 , piece_id=C2JSPN2QVSGSXG6ZW3ZMOFX5YFNN65XY7E62N5OEHOOMK3NSO4KQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.012963184 (524h17m26.475055667s) [noise=true , address=145.79.227.16:4910 , piece_id=BIHEJDH7RPI4UCPUNV453QHSEVCLA54PSNNQB2II37XZDRXFZNWQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.012994684 (524h17m26.475087167s) [noise=true , address=145.79.227.14:4840 , piece_id=B4ZVPRMVH7Y3TCWCLPO7DWRVMJL7VSI73CIT4PEHCMTIEURQKB2Q, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.016649956 (524h17m26.478742439s) [noise=true , address=145.79.227.11:4730 , piece_id=VO2DSM5LT43QAFPWOY3W3PVEJ44GZJWTCWG5QRQQFB363DFVFZDA, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.019248864 (524h17m26.481341347s) [noise=true , address=145.79.227.17:4969 , piece_id=R5HIP33QW5HLICARFWDUOW7I75YTDHZTJCIML5X52VKHLPL3J4YQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.075991585 (524h17m26.538084068s) [noise=true , address=145.79.227.4:4187 , piece_id=TQVPL6ZA6LHKAJVL6XWSYASZD3MCKKV3553CRFD72XWTVWY62STQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.118655146 (524h17m26.580747619s) [noise=true , address=145.79.227.14:4838 , piece_id=6K6A4CNEHGB3X6DEBLF5KHLUT4GE6BQBW6IE3RHBML2NWPN255AA, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.285934795 (524h17m26.748027278s) [noise=true , address=77.90.31.11:4555 , piece_id=U5RYZSD643KSUI4XLL26EAN6FQRPMG62GQMMCQRPEEG2XKSNBIZQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.744525465 (524h17m27.206617948s) [noise=true , address=45.91.116.9:4262 , piece_id=X53JZGRLZLX53UZLLCXXUQCFSQD2FO76X74OM45G26HNDU55UXXQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:42.218961989 (524h17m27.681054462s) [noise=true , address=45.91.116.11:4117 , piece_id=VH7ZJKSZ2NUXQJZE2IKNWP5PAL3TN2NNHD3ATZUQHGKKYMSSM3LQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:42.313638675 (524h17m27.775731158s) [noise=true , address=45.91.116.14:4189 , piece_id=UMQRQYPZ7SWUCO2N7JDCEVI5U5P36FDL3SUM6HIYJVWVOZTWNXQQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:42.434283559 (524h17m27.896376053s) [noise=true , address=45.91.116.8:4045 , piece_id=EDXBNLKKRZMT6MKK2KOIZM7DDKA3IHB6UMCXX4YKZF7JY2T3S5CQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:42.434321832 (524h17m27.896414305s) [seg_pos={0 0}, upload=461218]: Segment reached optimal threshold of 13 pieces. Oct 15 09:48:42.434605041 (524h17m27.896697534s) [noise=true , address=45.91.116.5:4299 , piece_id=4ARMXXE5LBRE2OZQEIITELKPTW4TISZWTD6XZGI4UJV7UCLUXAEQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX): protocol: expected piece hash; context canceled; EOF Oct 15 09:48:42.434666138 (524h17m27.896758621s) [noise=true , address=45.91.116.5:4299 , piece_id=4ARMXXE5LBRE2OZQEIITELKPTW4TISZWTD6XZGI4UJV7UCLUXAEQ, seg_pos={0 0}, upload=461218]: Long tail cancellation due to optimal threshold Oct 15 09:48:42.434692849 (524h17m27.896785432s) [noise=true , address=45.91.116.12:4226 , piece_id=DRXRGO7OXEKVS5ACL6FC7RKLGBBD6KNI7X6E5DM5D5VV5PEKN7FQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1): protocol: expected piece hash; context canceled; EOF Oct 15 09:48:42.434706184 (524h17m27.896798657s) [noise=true , address=45.91.116.12:4226 , piece_id=DRXRGO7OXEKVS5ACL6FC7RKLGBBD6KNI7X6E5DM5D5VV5PEKN7FQ, seg_pos={0 0}, upload=461218]: Long tail cancellation due to optimal threshold Oct 15 09:48:42.434718318 (524h17m27.896810911s) [noise=true , address=45.91.116.4:4940 , piece_id=3MCSUDYN73NIVJTGKPOVY6N3MALB3GBXI7I5NAX4HXPDHVMQ52BA, seg_pos={0 0}, upload=461218]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe): protocol: expected piece hash; context canceled; EOF Oct 15 09:48:42.434728447 (524h17m27.89682093s) [noise=true , address=45.91.116.4:4940 , piece_id=3MCSUDYN73NIVJTGKPOVY6N3MALB3GBXI7I5NAX4HXPDHVMQ52BA, seg_pos={0 0}, upload=461218]: Long tail cancellation due to optimal threshold Oct 15 09:48:42.434760458 (524h17m27.896852931s) [seg_pos={0 0}, upload=461218]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 15 09:48:42.434866900 (524h17m27.896959384s) [upload=461218]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 15 09:48:42.434873884 (524h17m27.896966367s) [upload=461218]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 15 09:48:42.760633614 (524h17m28.222726087s) [upload=461218]: Done uploading segments... Oct 15 09:48:42.760655435 (524h17m28.222747918s) [upload=461218]: Upload finished. err: <nil> Duration: 6.031990604s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...........................................X...X............] duration: 5.961124228s FAIL (2025-10-15 09:57:11 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, Piece ID: VH7ZJKSZ2NUXQJZE2IKNWP5PAL3TN2NNHD3ATZUQHGKKYMSSM3LQ) context deadline exceeded; piecestore: (Node ID: 1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N, Piece ID: UMQRQYPZ7SWUCO2N7JDCEVI5U5P36FDL3SUM6HIYJVWVOZTWNXQQ) context deadline exceeded ======LOGS====== Oct 15 09:48:38.801058027 (524h17m24.26315051s) [upload=461218]: Starting upload Oct 15 09:48:38.801138119 (524h17m24.263230592s) [upload=461218]: Done starting upload Oct 15 09:48:38.801288545 (524h17m24.263381028s) [upload=461218]: Uploading segments... Oct 15 09:48:38.801442509 (524h17m24.263535002s) [upload=461218]: Got next segment. Inline: false Oct 15 09:48:38.801486663 (524h17m24.263579136s) [upload=461218]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 15 09:48:38.801493245 (524h17m24.263585728s) [upload=461218]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 15 09:48:39.066131918 (524h17m24.528224401s) [seg_pos={0 0}, upload=461218]: Begin upload segment... Oct 15 09:48:39.066205549 (524h17m24.528298022s) [seg_pos={0 0}, upload=461218]: Done begin upload segment. Oct 15 09:48:39.066243090 (524h17m24.528335573s) [noise=true , address=45.91.116.5:4299 , piece_id=4ARMXXE5LBRE2OZQEIITELKPTW4TISZWTD6XZGI4UJV7UCLUXAEQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.066254431 (524h17m24.528346915s) [noise=true , address=45.91.116.11:4117 , piece_id=VH7ZJKSZ2NUXQJZE2IKNWP5PAL3TN2NNHD3ATZUQHGKKYMSSM3LQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.066874873 (524h17m24.528967356s) [noise=true , address=145.79.227.7:4294 , piece_id=H4ZNVIXORNO4MTXOVYDQZI4IJE4VXHZWC6JI35YKOR3UESSXJNTQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.067301676 (524h17m24.529394159s) [noise=true , address=45.91.116.9:4262 , piece_id=X53JZGRLZLX53UZLLCXXUQCFSQD2FO76X74OM45G26HNDU55UXXQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.067705735 (524h17m24.529798218s) [noise=true , address=145.79.227.4:4187 , piece_id=TQVPL6ZA6LHKAJVL6XWSYASZD3MCKKV3553CRFD72XWTVWY62STQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.069183369 (524h17m24.531275862s) [noise=true , address=145.79.227.5:4222 , piece_id=C2JSPN2QVSGSXG6ZW3ZMOFX5YFNN65XY7E62N5OEHOOMK3NSO4KQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.070534782 (524h17m24.532627265s) [noise=true , address=77.90.31.11:4555 , piece_id=U5RYZSD643KSUI4XLL26EAN6FQRPMG62GQMMCQRPEEG2XKSNBIZQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.072356982 (524h17m24.534449466s) [noise=true , address=145.79.227.11:4730 , piece_id=VO2DSM5LT43QAFPWOY3W3PVEJ44GZJWTCWG5QRQQFB363DFVFZDA, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.073961167 (524h17m24.53605364s) [noise=true , address=45.91.116.4:4940 , piece_id=3MCSUDYN73NIVJTGKPOVY6N3MALB3GBXI7I5NAX4HXPDHVMQ52BA, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.075716960 (524h17m24.537809433s) [noise=true , address=145.79.227.16:4910 , piece_id=BIHEJDH7RPI4UCPUNV453QHSEVCLA54PSNNQB2II37XZDRXFZNWQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.079266419 (524h17m24.541358902s) [noise=true , address=145.79.227.14:4838 , piece_id=6K6A4CNEHGB3X6DEBLF5KHLUT4GE6BQBW6IE3RHBML2NWPN255AA, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.080943032 (524h17m24.543035505s) [noise=true , address=45.91.116.12:4226 , piece_id=DRXRGO7OXEKVS5ACL6FC7RKLGBBD6KNI7X6E5DM5D5VV5PEKN7FQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.081340479 (524h17m24.543432962s) [noise=true , address=45.91.116.8:4045 , piece_id=EDXBNLKKRZMT6MKK2KOIZM7DDKA3IHB6UMCXX4YKZF7JY2T3S5CQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.083616223 (524h17m24.545708706s) [noise=true , address=145.79.227.14:4840 , piece_id=B4ZVPRMVH7Y3TCWCLPO7DWRVMJL7VSI73CIT4PEHCMTIEURQKB2Q, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.083633946 (524h17m24.545726429s) [noise=true , address=45.91.116.14:4189 , piece_id=UMQRQYPZ7SWUCO2N7JDCEVI5U5P36FDL3SUM6HIYJVWVOZTWNXQQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.085493247 (524h17m24.54758573s) [noise=true , address=145.79.227.17:4969 , piece_id=R5HIP33QW5HLICARFWDUOW7I75YTDHZTJCIML5X52VKHLPL3J4YQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:40.727263388 (524h17m26.189355871s) [upload=461218]: Next returned nil segment Oct 15 09:48:40.727310076 (524h17m26.189402549s) [upload=461218]: Waiting for error group managing segments... Oct 15 09:48:41.004514306 (524h17m26.466606789s) [noise=true , address=145.79.227.7:4294 , piece_id=H4ZNVIXORNO4MTXOVYDQZI4IJE4VXHZWC6JI35YKOR3UESSXJNTQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.006116557 (524h17m26.46820904s) [noise=true , address=145.79.227.5:4222 , piece_id=C2JSPN2QVSGSXG6ZW3ZMOFX5YFNN65XY7E62N5OEHOOMK3NSO4KQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.012963184 (524h17m26.475055667s) [noise=true , address=145.79.227.16:4910 , piece_id=BIHEJDH7RPI4UCPUNV453QHSEVCLA54PSNNQB2II37XZDRXFZNWQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.012994684 (524h17m26.475087167s) [noise=true , address=145.79.227.14:4840 , piece_id=B4ZVPRMVH7Y3TCWCLPO7DWRVMJL7VSI73CIT4PEHCMTIEURQKB2Q, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.016649956 (524h17m26.478742439s) [noise=true , address=145.79.227.11:4730 , piece_id=VO2DSM5LT43QAFPWOY3W3PVEJ44GZJWTCWG5QRQQFB363DFVFZDA, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.019248864 (524h17m26.481341347s) [noise=true , address=145.79.227.17:4969 , piece_id=R5HIP33QW5HLICARFWDUOW7I75YTDHZTJCIML5X52VKHLPL3J4YQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.075991585 (524h17m26.538084068s) [noise=true , address=145.79.227.4:4187 , piece_id=TQVPL6ZA6LHKAJVL6XWSYASZD3MCKKV3553CRFD72XWTVWY62STQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.118655146 (524h17m26.580747619s) [noise=true , address=145.79.227.14:4838 , piece_id=6K6A4CNEHGB3X6DEBLF5KHLUT4GE6BQBW6IE3RHBML2NWPN255AA, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.285934795 (524h17m26.748027278s) [noise=true , address=77.90.31.11:4555 , piece_id=U5RYZSD643KSUI4XLL26EAN6FQRPMG62GQMMCQRPEEG2XKSNBIZQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.744525465 (524h17m27.206617948s) [noise=true , address=45.91.116.9:4262 , piece_id=X53JZGRLZLX53UZLLCXXUQCFSQD2FO76X74OM45G26HNDU55UXXQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:42.218961989 (524h17m27.681054462s) [noise=true , address=45.91.116.11:4117 , piece_id=VH7ZJKSZ2NUXQJZE2IKNWP5PAL3TN2NNHD3ATZUQHGKKYMSSM3LQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:42.313638675 (524h17m27.775731158s) [noise=true , address=45.91.116.14:4189 , piece_id=UMQRQYPZ7SWUCO2N7JDCEVI5U5P36FDL3SUM6HIYJVWVOZTWNXQQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:42.434283559 (524h17m27.896376053s) [noise=true , address=45.91.116.8:4045 , piece_id=EDXBNLKKRZMT6MKK2KOIZM7DDKA3IHB6UMCXX4YKZF7JY2T3S5CQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:42.434321832 (524h17m27.896414305s) [seg_pos={0 0}, upload=461218]: Segment reached optimal threshold of 13 pieces. Oct 15 09:48:42.434605041 (524h17m27.896697534s) [noise=true , address=45.91.116.5:4299 , piece_id=4ARMXXE5LBRE2OZQEIITELKPTW4TISZWTD6XZGI4UJV7UCLUXAEQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX): protocol: expected piece hash; context canceled; EOF Oct 15 09:48:42.434666138 (524h17m27.896758621s) [noise=true , address=45.91.116.5:4299 , piece_id=4ARMXXE5LBRE2OZQEIITELKPTW4TISZWTD6XZGI4UJV7UCLUXAEQ, seg_pos={0 0}, upload=461218]: Long tail cancellation due to optimal threshold Oct 15 09:48:42.434692849 (524h17m27.896785432s) [noise=true , address=45.91.116.12:4226 , piece_id=DRXRGO7OXEKVS5ACL6FC7RKLGBBD6KNI7X6E5DM5D5VV5PEKN7FQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1): protocol: expected piece hash; context canceled; EOF Oct 15 09:48:42.434706184 (524h17m27.896798657s) [noise=true , address=45.91.116.12:4226 , piece_id=DRXRGO7OXEKVS5ACL6FC7RKLGBBD6KNI7X6E5DM5D5VV5PEKN7FQ, seg_pos={0 0}, upload=461218]: Long tail cancellation due to optimal threshold Oct 15 09:48:42.434718318 (524h17m27.896810911s) [noise=true , address=45.91.116.4:4940 , piece_id=3MCSUDYN73NIVJTGKPOVY6N3MALB3GBXI7I5NAX4HXPDHVMQ52BA, seg_pos={0 0}, upload=461218]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe): protocol: expected piece hash; context canceled; EOF Oct 15 09:48:42.434728447 (524h17m27.89682093s) [noise=true , address=45.91.116.4:4940 , piece_id=3MCSUDYN73NIVJTGKPOVY6N3MALB3GBXI7I5NAX4HXPDHVMQ52BA, seg_pos={0 0}, upload=461218]: Long tail cancellation due to optimal threshold Oct 15 09:48:42.434760458 (524h17m27.896852931s) [seg_pos={0 0}, upload=461218]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 15 09:48:42.434866900 (524h17m27.896959384s) [upload=461218]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 15 09:48:42.434873884 (524h17m27.896966367s) [upload=461218]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 15 09:48:42.760633614 (524h17m28.222726087s) [upload=461218]: Done uploading segments... Oct 15 09:48:42.760655435 (524h17m28.222747918s) [upload=461218]: Upload finished. err: <nil> Duration: 5.96114632s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [............................................X...X...........] duration: 6.083740048s FAIL (2025-10-15 09:56:42 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, Piece ID: VH7ZJKSZ2NUXQJZE2IKNWP5PAL3TN2NNHD3ATZUQHGKKYMSSM3LQ) context deadline exceeded; piecestore: (Node ID: 1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N, Piece ID: UMQRQYPZ7SWUCO2N7JDCEVI5U5P36FDL3SUM6HIYJVWVOZTWNXQQ) context deadline exceeded ======LOGS====== Oct 15 09:48:38.801058027 (524h17m24.26315051s) [upload=461218]: Starting upload Oct 15 09:48:38.801138119 (524h17m24.263230592s) [upload=461218]: Done starting upload Oct 15 09:48:38.801288545 (524h17m24.263381028s) [upload=461218]: Uploading segments... Oct 15 09:48:38.801442509 (524h17m24.263535002s) [upload=461218]: Got next segment. Inline: false Oct 15 09:48:38.801486663 (524h17m24.263579136s) [upload=461218]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 15 09:48:38.801493245 (524h17m24.263585728s) [upload=461218]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 15 09:48:39.066131918 (524h17m24.528224401s) [seg_pos={0 0}, upload=461218]: Begin upload segment... Oct 15 09:48:39.066205549 (524h17m24.528298022s) [seg_pos={0 0}, upload=461218]: Done begin upload segment. Oct 15 09:48:39.066243090 (524h17m24.528335573s) [noise=true , address=45.91.116.5:4299 , piece_id=4ARMXXE5LBRE2OZQEIITELKPTW4TISZWTD6XZGI4UJV7UCLUXAEQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.066254431 (524h17m24.528346915s) [noise=true , address=45.91.116.11:4117 , piece_id=VH7ZJKSZ2NUXQJZE2IKNWP5PAL3TN2NNHD3ATZUQHGKKYMSSM3LQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.066874873 (524h17m24.528967356s) [noise=true , address=145.79.227.7:4294 , piece_id=H4ZNVIXORNO4MTXOVYDQZI4IJE4VXHZWC6JI35YKOR3UESSXJNTQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.067301676 (524h17m24.529394159s) [noise=true , address=45.91.116.9:4262 , piece_id=X53JZGRLZLX53UZLLCXXUQCFSQD2FO76X74OM45G26HNDU55UXXQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.067705735 (524h17m24.529798218s) [noise=true , address=145.79.227.4:4187 , piece_id=TQVPL6ZA6LHKAJVL6XWSYASZD3MCKKV3553CRFD72XWTVWY62STQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.069183369 (524h17m24.531275862s) [noise=true , address=145.79.227.5:4222 , piece_id=C2JSPN2QVSGSXG6ZW3ZMOFX5YFNN65XY7E62N5OEHOOMK3NSO4KQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.070534782 (524h17m24.532627265s) [noise=true , address=77.90.31.11:4555 , piece_id=U5RYZSD643KSUI4XLL26EAN6FQRPMG62GQMMCQRPEEG2XKSNBIZQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.072356982 (524h17m24.534449466s) [noise=true , address=145.79.227.11:4730 , piece_id=VO2DSM5LT43QAFPWOY3W3PVEJ44GZJWTCWG5QRQQFB363DFVFZDA, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.073961167 (524h17m24.53605364s) [noise=true , address=45.91.116.4:4940 , piece_id=3MCSUDYN73NIVJTGKPOVY6N3MALB3GBXI7I5NAX4HXPDHVMQ52BA, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.075716960 (524h17m24.537809433s) [noise=true , address=145.79.227.16:4910 , piece_id=BIHEJDH7RPI4UCPUNV453QHSEVCLA54PSNNQB2II37XZDRXFZNWQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.079266419 (524h17m24.541358902s) [noise=true , address=145.79.227.14:4838 , piece_id=6K6A4CNEHGB3X6DEBLF5KHLUT4GE6BQBW6IE3RHBML2NWPN255AA, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.080943032 (524h17m24.543035505s) [noise=true , address=45.91.116.12:4226 , piece_id=DRXRGO7OXEKVS5ACL6FC7RKLGBBD6KNI7X6E5DM5D5VV5PEKN7FQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.081340479 (524h17m24.543432962s) [noise=true , address=45.91.116.8:4045 , piece_id=EDXBNLKKRZMT6MKK2KOIZM7DDKA3IHB6UMCXX4YKZF7JY2T3S5CQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.083616223 (524h17m24.545708706s) [noise=true , address=145.79.227.14:4840 , piece_id=B4ZVPRMVH7Y3TCWCLPO7DWRVMJL7VSI73CIT4PEHCMTIEURQKB2Q, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.083633946 (524h17m24.545726429s) [noise=true , address=45.91.116.14:4189 , piece_id=UMQRQYPZ7SWUCO2N7JDCEVI5U5P36FDL3SUM6HIYJVWVOZTWNXQQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.085493247 (524h17m24.54758573s) [noise=true , address=145.79.227.17:4969 , piece_id=R5HIP33QW5HLICARFWDUOW7I75YTDHZTJCIML5X52VKHLPL3J4YQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:40.727263388 (524h17m26.189355871s) [upload=461218]: Next returned nil segment Oct 15 09:48:40.727310076 (524h17m26.189402549s) [upload=461218]: Waiting for error group managing segments... Oct 15 09:48:41.004514306 (524h17m26.466606789s) [noise=true , address=145.79.227.7:4294 , piece_id=H4ZNVIXORNO4MTXOVYDQZI4IJE4VXHZWC6JI35YKOR3UESSXJNTQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.006116557 (524h17m26.46820904s) [noise=true , address=145.79.227.5:4222 , piece_id=C2JSPN2QVSGSXG6ZW3ZMOFX5YFNN65XY7E62N5OEHOOMK3NSO4KQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.012963184 (524h17m26.475055667s) [noise=true , address=145.79.227.16:4910 , piece_id=BIHEJDH7RPI4UCPUNV453QHSEVCLA54PSNNQB2II37XZDRXFZNWQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.012994684 (524h17m26.475087167s) [noise=true , address=145.79.227.14:4840 , piece_id=B4ZVPRMVH7Y3TCWCLPO7DWRVMJL7VSI73CIT4PEHCMTIEURQKB2Q, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.016649956 (524h17m26.478742439s) [noise=true , address=145.79.227.11:4730 , piece_id=VO2DSM5LT43QAFPWOY3W3PVEJ44GZJWTCWG5QRQQFB363DFVFZDA, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.019248864 (524h17m26.481341347s) [noise=true , address=145.79.227.17:4969 , piece_id=R5HIP33QW5HLICARFWDUOW7I75YTDHZTJCIML5X52VKHLPL3J4YQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.075991585 (524h17m26.538084068s) [noise=true , address=145.79.227.4:4187 , piece_id=TQVPL6ZA6LHKAJVL6XWSYASZD3MCKKV3553CRFD72XWTVWY62STQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.118655146 (524h17m26.580747619s) [noise=true , address=145.79.227.14:4838 , piece_id=6K6A4CNEHGB3X6DEBLF5KHLUT4GE6BQBW6IE3RHBML2NWPN255AA, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.285934795 (524h17m26.748027278s) [noise=true , address=77.90.31.11:4555 , piece_id=U5RYZSD643KSUI4XLL26EAN6FQRPMG62GQMMCQRPEEG2XKSNBIZQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.744525465 (524h17m27.206617948s) [noise=true , address=45.91.116.9:4262 , piece_id=X53JZGRLZLX53UZLLCXXUQCFSQD2FO76X74OM45G26HNDU55UXXQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:42.218961989 (524h17m27.681054462s) [noise=true , address=45.91.116.11:4117 , piece_id=VH7ZJKSZ2NUXQJZE2IKNWP5PAL3TN2NNHD3ATZUQHGKKYMSSM3LQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:42.313638675 (524h17m27.775731158s) [noise=true , address=45.91.116.14:4189 , piece_id=UMQRQYPZ7SWUCO2N7JDCEVI5U5P36FDL3SUM6HIYJVWVOZTWNXQQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:42.434283559 (524h17m27.896376053s) [noise=true , address=45.91.116.8:4045 , piece_id=EDXBNLKKRZMT6MKK2KOIZM7DDKA3IHB6UMCXX4YKZF7JY2T3S5CQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:42.434321832 (524h17m27.896414305s) [seg_pos={0 0}, upload=461218]: Segment reached optimal threshold of 13 pieces. Oct 15 09:48:42.434605041 (524h17m27.896697534s) [noise=true , address=45.91.116.5:4299 , piece_id=4ARMXXE5LBRE2OZQEIITELKPTW4TISZWTD6XZGI4UJV7UCLUXAEQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX): protocol: expected piece hash; context canceled; EOF Oct 15 09:48:42.434666138 (524h17m27.896758621s) [noise=true , address=45.91.116.5:4299 , piece_id=4ARMXXE5LBRE2OZQEIITELKPTW4TISZWTD6XZGI4UJV7UCLUXAEQ, seg_pos={0 0}, upload=461218]: Long tail cancellation due to optimal threshold Oct 15 09:48:42.434692849 (524h17m27.896785432s) [noise=true , address=45.91.116.12:4226 , piece_id=DRXRGO7OXEKVS5ACL6FC7RKLGBBD6KNI7X6E5DM5D5VV5PEKN7FQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1): protocol: expected piece hash; context canceled; EOF Oct 15 09:48:42.434706184 (524h17m27.896798657s) [noise=true , address=45.91.116.12:4226 , piece_id=DRXRGO7OXEKVS5ACL6FC7RKLGBBD6KNI7X6E5DM5D5VV5PEKN7FQ, seg_pos={0 0}, upload=461218]: Long tail cancellation due to optimal threshold Oct 15 09:48:42.434718318 (524h17m27.896810911s) [noise=true , address=45.91.116.4:4940 , piece_id=3MCSUDYN73NIVJTGKPOVY6N3MALB3GBXI7I5NAX4HXPDHVMQ52BA, seg_pos={0 0}, upload=461218]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe): protocol: expected piece hash; context canceled; EOF Oct 15 09:48:42.434728447 (524h17m27.89682093s) [noise=true , address=45.91.116.4:4940 , piece_id=3MCSUDYN73NIVJTGKPOVY6N3MALB3GBXI7I5NAX4HXPDHVMQ52BA, seg_pos={0 0}, upload=461218]: Long tail cancellation due to optimal threshold Oct 15 09:48:42.434760458 (524h17m27.896852931s) [seg_pos={0 0}, upload=461218]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 15 09:48:42.434866900 (524h17m27.896959384s) [upload=461218]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 15 09:48:42.434873884 (524h17m27.896966367s) [upload=461218]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 15 09:48:42.760633614 (524h17m28.222726087s) [upload=461218]: Done uploading segments... Oct 15 09:48:42.760655435 (524h17m28.222747918s) [upload=461218]: Upload finished. err: <nil> Duration: 6.083775635s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.............................................X...X..........] duration: 6.713017736s FAIL (2025-10-15 09:55:52 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, Piece ID: VH7ZJKSZ2NUXQJZE2IKNWP5PAL3TN2NNHD3ATZUQHGKKYMSSM3LQ) context deadline exceeded; piecestore: (Node ID: 1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N, Piece ID: UMQRQYPZ7SWUCO2N7JDCEVI5U5P36FDL3SUM6HIYJVWVOZTWNXQQ) context deadline exceeded ======LOGS====== Oct 15 09:48:38.801058027 (524h17m24.26315051s) [upload=461218]: Starting upload Oct 15 09:48:38.801138119 (524h17m24.263230592s) [upload=461218]: Done starting upload Oct 15 09:48:38.801288545 (524h17m24.263381028s) [upload=461218]: Uploading segments... Oct 15 09:48:38.801442509 (524h17m24.263535002s) [upload=461218]: Got next segment. Inline: false Oct 15 09:48:38.801486663 (524h17m24.263579136s) [upload=461218]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 15 09:48:38.801493245 (524h17m24.263585728s) [upload=461218]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 15 09:48:39.066131918 (524h17m24.528224401s) [seg_pos={0 0}, upload=461218]: Begin upload segment... Oct 15 09:48:39.066205549 (524h17m24.528298022s) [seg_pos={0 0}, upload=461218]: Done begin upload segment. Oct 15 09:48:39.066243090 (524h17m24.528335573s) [noise=true , address=45.91.116.5:4299 , piece_id=4ARMXXE5LBRE2OZQEIITELKPTW4TISZWTD6XZGI4UJV7UCLUXAEQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.066254431 (524h17m24.528346915s) [noise=true , address=45.91.116.11:4117 , piece_id=VH7ZJKSZ2NUXQJZE2IKNWP5PAL3TN2NNHD3ATZUQHGKKYMSSM3LQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.066874873 (524h17m24.528967356s) [noise=true , address=145.79.227.7:4294 , piece_id=H4ZNVIXORNO4MTXOVYDQZI4IJE4VXHZWC6JI35YKOR3UESSXJNTQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.067301676 (524h17m24.529394159s) [noise=true , address=45.91.116.9:4262 , piece_id=X53JZGRLZLX53UZLLCXXUQCFSQD2FO76X74OM45G26HNDU55UXXQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.067705735 (524h17m24.529798218s) [noise=true , address=145.79.227.4:4187 , piece_id=TQVPL6ZA6LHKAJVL6XWSYASZD3MCKKV3553CRFD72XWTVWY62STQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.069183369 (524h17m24.531275862s) [noise=true , address=145.79.227.5:4222 , piece_id=C2JSPN2QVSGSXG6ZW3ZMOFX5YFNN65XY7E62N5OEHOOMK3NSO4KQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.070534782 (524h17m24.532627265s) [noise=true , address=77.90.31.11:4555 , piece_id=U5RYZSD643KSUI4XLL26EAN6FQRPMG62GQMMCQRPEEG2XKSNBIZQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.072356982 (524h17m24.534449466s) [noise=true , address=145.79.227.11:4730 , piece_id=VO2DSM5LT43QAFPWOY3W3PVEJ44GZJWTCWG5QRQQFB363DFVFZDA, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.073961167 (524h17m24.53605364s) [noise=true , address=45.91.116.4:4940 , piece_id=3MCSUDYN73NIVJTGKPOVY6N3MALB3GBXI7I5NAX4HXPDHVMQ52BA, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.075716960 (524h17m24.537809433s) [noise=true , address=145.79.227.16:4910 , piece_id=BIHEJDH7RPI4UCPUNV453QHSEVCLA54PSNNQB2II37XZDRXFZNWQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.079266419 (524h17m24.541358902s) [noise=true , address=145.79.227.14:4838 , piece_id=6K6A4CNEHGB3X6DEBLF5KHLUT4GE6BQBW6IE3RHBML2NWPN255AA, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.080943032 (524h17m24.543035505s) [noise=true , address=45.91.116.12:4226 , piece_id=DRXRGO7OXEKVS5ACL6FC7RKLGBBD6KNI7X6E5DM5D5VV5PEKN7FQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.081340479 (524h17m24.543432962s) [noise=true , address=45.91.116.8:4045 , piece_id=EDXBNLKKRZMT6MKK2KOIZM7DDKA3IHB6UMCXX4YKZF7JY2T3S5CQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.083616223 (524h17m24.545708706s) [noise=true , address=145.79.227.14:4840 , piece_id=B4ZVPRMVH7Y3TCWCLPO7DWRVMJL7VSI73CIT4PEHCMTIEURQKB2Q, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.083633946 (524h17m24.545726429s) [noise=true , address=45.91.116.14:4189 , piece_id=UMQRQYPZ7SWUCO2N7JDCEVI5U5P36FDL3SUM6HIYJVWVOZTWNXQQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.085493247 (524h17m24.54758573s) [noise=true , address=145.79.227.17:4969 , piece_id=R5HIP33QW5HLICARFWDUOW7I75YTDHZTJCIML5X52VKHLPL3J4YQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:40.727263388 (524h17m26.189355871s) [upload=461218]: Next returned nil segment Oct 15 09:48:40.727310076 (524h17m26.189402549s) [upload=461218]: Waiting for error group managing segments... Oct 15 09:48:41.004514306 (524h17m26.466606789s) [noise=true , address=145.79.227.7:4294 , piece_id=H4ZNVIXORNO4MTXOVYDQZI4IJE4VXHZWC6JI35YKOR3UESSXJNTQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.006116557 (524h17m26.46820904s) [noise=true , address=145.79.227.5:4222 , piece_id=C2JSPN2QVSGSXG6ZW3ZMOFX5YFNN65XY7E62N5OEHOOMK3NSO4KQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.012963184 (524h17m26.475055667s) [noise=true , address=145.79.227.16:4910 , piece_id=BIHEJDH7RPI4UCPUNV453QHSEVCLA54PSNNQB2II37XZDRXFZNWQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.012994684 (524h17m26.475087167s) [noise=true , address=145.79.227.14:4840 , piece_id=B4ZVPRMVH7Y3TCWCLPO7DWRVMJL7VSI73CIT4PEHCMTIEURQKB2Q, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.016649956 (524h17m26.478742439s) [noise=true , address=145.79.227.11:4730 , piece_id=VO2DSM5LT43QAFPWOY3W3PVEJ44GZJWTCWG5QRQQFB363DFVFZDA, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.019248864 (524h17m26.481341347s) [noise=true , address=145.79.227.17:4969 , piece_id=R5HIP33QW5HLICARFWDUOW7I75YTDHZTJCIML5X52VKHLPL3J4YQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.075991585 (524h17m26.538084068s) [noise=true , address=145.79.227.4:4187 , piece_id=TQVPL6ZA6LHKAJVL6XWSYASZD3MCKKV3553CRFD72XWTVWY62STQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.118655146 (524h17m26.580747619s) [noise=true , address=145.79.227.14:4838 , piece_id=6K6A4CNEHGB3X6DEBLF5KHLUT4GE6BQBW6IE3RHBML2NWPN255AA, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.285934795 (524h17m26.748027278s) [noise=true , address=77.90.31.11:4555 , piece_id=U5RYZSD643KSUI4XLL26EAN6FQRPMG62GQMMCQRPEEG2XKSNBIZQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.744525465 (524h17m27.206617948s) [noise=true , address=45.91.116.9:4262 , piece_id=X53JZGRLZLX53UZLLCXXUQCFSQD2FO76X74OM45G26HNDU55UXXQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:42.218961989 (524h17m27.681054462s) [noise=true , address=45.91.116.11:4117 , piece_id=VH7ZJKSZ2NUXQJZE2IKNWP5PAL3TN2NNHD3ATZUQHGKKYMSSM3LQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:42.313638675 (524h17m27.775731158s) [noise=true , address=45.91.116.14:4189 , piece_id=UMQRQYPZ7SWUCO2N7JDCEVI5U5P36FDL3SUM6HIYJVWVOZTWNXQQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:42.434283559 (524h17m27.896376053s) [noise=true , address=45.91.116.8:4045 , piece_id=EDXBNLKKRZMT6MKK2KOIZM7DDKA3IHB6UMCXX4YKZF7JY2T3S5CQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:42.434321832 (524h17m27.896414305s) [seg_pos={0 0}, upload=461218]: Segment reached optimal threshold of 13 pieces. Oct 15 09:48:42.434605041 (524h17m27.896697534s) [noise=true , address=45.91.116.5:4299 , piece_id=4ARMXXE5LBRE2OZQEIITELKPTW4TISZWTD6XZGI4UJV7UCLUXAEQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX): protocol: expected piece hash; context canceled; EOF Oct 15 09:48:42.434666138 (524h17m27.896758621s) [noise=true , address=45.91.116.5:4299 , piece_id=4ARMXXE5LBRE2OZQEIITELKPTW4TISZWTD6XZGI4UJV7UCLUXAEQ, seg_pos={0 0}, upload=461218]: Long tail cancellation due to optimal threshold Oct 15 09:48:42.434692849 (524h17m27.896785432s) [noise=true , address=45.91.116.12:4226 , piece_id=DRXRGO7OXEKVS5ACL6FC7RKLGBBD6KNI7X6E5DM5D5VV5PEKN7FQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1): protocol: expected piece hash; context canceled; EOF Oct 15 09:48:42.434706184 (524h17m27.896798657s) [noise=true , address=45.91.116.12:4226 , piece_id=DRXRGO7OXEKVS5ACL6FC7RKLGBBD6KNI7X6E5DM5D5VV5PEKN7FQ, seg_pos={0 0}, upload=461218]: Long tail cancellation due to optimal threshold Oct 15 09:48:42.434718318 (524h17m27.896810911s) [noise=true , address=45.91.116.4:4940 , piece_id=3MCSUDYN73NIVJTGKPOVY6N3MALB3GBXI7I5NAX4HXPDHVMQ52BA, seg_pos={0 0}, upload=461218]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe): protocol: expected piece hash; context canceled; EOF Oct 15 09:48:42.434728447 (524h17m27.89682093s) [noise=true , address=45.91.116.4:4940 , piece_id=3MCSUDYN73NIVJTGKPOVY6N3MALB3GBXI7I5NAX4HXPDHVMQ52BA, seg_pos={0 0}, upload=461218]: Long tail cancellation due to optimal threshold Oct 15 09:48:42.434760458 (524h17m27.896852931s) [seg_pos={0 0}, upload=461218]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 15 09:48:42.434866900 (524h17m27.896959384s) [upload=461218]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 15 09:48:42.434873884 (524h17m27.896966367s) [upload=461218]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 15 09:48:42.760633614 (524h17m28.222726087s) [upload=461218]: Done uploading segments... Oct 15 09:48:42.760655435 (524h17m28.222747918s) [upload=461218]: Upload finished. err: <nil> Duration: 6.713050119s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..............................................X...X.........] duration: 7.700552496s FAIL (2025-10-15 09:55:08 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, Piece ID: VH7ZJKSZ2NUXQJZE2IKNWP5PAL3TN2NNHD3ATZUQHGKKYMSSM3LQ) context deadline exceeded; piecestore: (Node ID: 1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N, Piece ID: UMQRQYPZ7SWUCO2N7JDCEVI5U5P36FDL3SUM6HIYJVWVOZTWNXQQ) context deadline exceeded ======LOGS====== Oct 15 09:48:38.801058027 (524h17m24.26315051s) [upload=461218]: Starting upload Oct 15 09:48:38.801138119 (524h17m24.263230592s) [upload=461218]: Done starting upload Oct 15 09:48:38.801288545 (524h17m24.263381028s) [upload=461218]: Uploading segments... Oct 15 09:48:38.801442509 (524h17m24.263535002s) [upload=461218]: Got next segment. Inline: false Oct 15 09:48:38.801486663 (524h17m24.263579136s) [upload=461218]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 15 09:48:38.801493245 (524h17m24.263585728s) [upload=461218]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 15 09:48:39.066131918 (524h17m24.528224401s) [seg_pos={0 0}, upload=461218]: Begin upload segment... Oct 15 09:48:39.066205549 (524h17m24.528298022s) [seg_pos={0 0}, upload=461218]: Done begin upload segment. Oct 15 09:48:39.066243090 (524h17m24.528335573s) [noise=true , address=45.91.116.5:4299 , piece_id=4ARMXXE5LBRE2OZQEIITELKPTW4TISZWTD6XZGI4UJV7UCLUXAEQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.066254431 (524h17m24.528346915s) [noise=true , address=45.91.116.11:4117 , piece_id=VH7ZJKSZ2NUXQJZE2IKNWP5PAL3TN2NNHD3ATZUQHGKKYMSSM3LQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.066874873 (524h17m24.528967356s) [noise=true , address=145.79.227.7:4294 , piece_id=H4ZNVIXORNO4MTXOVYDQZI4IJE4VXHZWC6JI35YKOR3UESSXJNTQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.067301676 (524h17m24.529394159s) [noise=true , address=45.91.116.9:4262 , piece_id=X53JZGRLZLX53UZLLCXXUQCFSQD2FO76X74OM45G26HNDU55UXXQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.067705735 (524h17m24.529798218s) [noise=true , address=145.79.227.4:4187 , piece_id=TQVPL6ZA6LHKAJVL6XWSYASZD3MCKKV3553CRFD72XWTVWY62STQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.069183369 (524h17m24.531275862s) [noise=true , address=145.79.227.5:4222 , piece_id=C2JSPN2QVSGSXG6ZW3ZMOFX5YFNN65XY7E62N5OEHOOMK3NSO4KQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.070534782 (524h17m24.532627265s) [noise=true , address=77.90.31.11:4555 , piece_id=U5RYZSD643KSUI4XLL26EAN6FQRPMG62GQMMCQRPEEG2XKSNBIZQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.072356982 (524h17m24.534449466s) [noise=true , address=145.79.227.11:4730 , piece_id=VO2DSM5LT43QAFPWOY3W3PVEJ44GZJWTCWG5QRQQFB363DFVFZDA, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.073961167 (524h17m24.53605364s) [noise=true , address=45.91.116.4:4940 , piece_id=3MCSUDYN73NIVJTGKPOVY6N3MALB3GBXI7I5NAX4HXPDHVMQ52BA, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.075716960 (524h17m24.537809433s) [noise=true , address=145.79.227.16:4910 , piece_id=BIHEJDH7RPI4UCPUNV453QHSEVCLA54PSNNQB2II37XZDRXFZNWQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.079266419 (524h17m24.541358902s) [noise=true , address=145.79.227.14:4838 , piece_id=6K6A4CNEHGB3X6DEBLF5KHLUT4GE6BQBW6IE3RHBML2NWPN255AA, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.080943032 (524h17m24.543035505s) [noise=true , address=45.91.116.12:4226 , piece_id=DRXRGO7OXEKVS5ACL6FC7RKLGBBD6KNI7X6E5DM5D5VV5PEKN7FQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.081340479 (524h17m24.543432962s) [noise=true , address=45.91.116.8:4045 , piece_id=EDXBNLKKRZMT6MKK2KOIZM7DDKA3IHB6UMCXX4YKZF7JY2T3S5CQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.083616223 (524h17m24.545708706s) [noise=true , address=145.79.227.14:4840 , piece_id=B4ZVPRMVH7Y3TCWCLPO7DWRVMJL7VSI73CIT4PEHCMTIEURQKB2Q, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.083633946 (524h17m24.545726429s) [noise=true , address=45.91.116.14:4189 , piece_id=UMQRQYPZ7SWUCO2N7JDCEVI5U5P36FDL3SUM6HIYJVWVOZTWNXQQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.085493247 (524h17m24.54758573s) [noise=true , address=145.79.227.17:4969 , piece_id=R5HIP33QW5HLICARFWDUOW7I75YTDHZTJCIML5X52VKHLPL3J4YQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:40.727263388 (524h17m26.189355871s) [upload=461218]: Next returned nil segment Oct 15 09:48:40.727310076 (524h17m26.189402549s) [upload=461218]: Waiting for error group managing segments... Oct 15 09:48:41.004514306 (524h17m26.466606789s) [noise=true , address=145.79.227.7:4294 , piece_id=H4ZNVIXORNO4MTXOVYDQZI4IJE4VXHZWC6JI35YKOR3UESSXJNTQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.006116557 (524h17m26.46820904s) [noise=true , address=145.79.227.5:4222 , piece_id=C2JSPN2QVSGSXG6ZW3ZMOFX5YFNN65XY7E62N5OEHOOMK3NSO4KQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.012963184 (524h17m26.475055667s) [noise=true , address=145.79.227.16:4910 , piece_id=BIHEJDH7RPI4UCPUNV453QHSEVCLA54PSNNQB2II37XZDRXFZNWQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.012994684 (524h17m26.475087167s) [noise=true , address=145.79.227.14:4840 , piece_id=B4ZVPRMVH7Y3TCWCLPO7DWRVMJL7VSI73CIT4PEHCMTIEURQKB2Q, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.016649956 (524h17m26.478742439s) [noise=true , address=145.79.227.11:4730 , piece_id=VO2DSM5LT43QAFPWOY3W3PVEJ44GZJWTCWG5QRQQFB363DFVFZDA, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.019248864 (524h17m26.481341347s) [noise=true , address=145.79.227.17:4969 , piece_id=R5HIP33QW5HLICARFWDUOW7I75YTDHZTJCIML5X52VKHLPL3J4YQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.075991585 (524h17m26.538084068s) [noise=true , address=145.79.227.4:4187 , piece_id=TQVPL6ZA6LHKAJVL6XWSYASZD3MCKKV3553CRFD72XWTVWY62STQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.118655146 (524h17m26.580747619s) [noise=true , address=145.79.227.14:4838 , piece_id=6K6A4CNEHGB3X6DEBLF5KHLUT4GE6BQBW6IE3RHBML2NWPN255AA, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.285934795 (524h17m26.748027278s) [noise=true , address=77.90.31.11:4555 , piece_id=U5RYZSD643KSUI4XLL26EAN6FQRPMG62GQMMCQRPEEG2XKSNBIZQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.744525465 (524h17m27.206617948s) [noise=true , address=45.91.116.9:4262 , piece_id=X53JZGRLZLX53UZLLCXXUQCFSQD2FO76X74OM45G26HNDU55UXXQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:42.218961989 (524h17m27.681054462s) [noise=true , address=45.91.116.11:4117 , piece_id=VH7ZJKSZ2NUXQJZE2IKNWP5PAL3TN2NNHD3ATZUQHGKKYMSSM3LQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:42.313638675 (524h17m27.775731158s) [noise=true , address=45.91.116.14:4189 , piece_id=UMQRQYPZ7SWUCO2N7JDCEVI5U5P36FDL3SUM6HIYJVWVOZTWNXQQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:42.434283559 (524h17m27.896376053s) [noise=true , address=45.91.116.8:4045 , piece_id=EDXBNLKKRZMT6MKK2KOIZM7DDKA3IHB6UMCXX4YKZF7JY2T3S5CQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:42.434321832 (524h17m27.896414305s) [seg_pos={0 0}, upload=461218]: Segment reached optimal threshold of 13 pieces. Oct 15 09:48:42.434605041 (524h17m27.896697534s) [noise=true , address=45.91.116.5:4299 , piece_id=4ARMXXE5LBRE2OZQEIITELKPTW4TISZWTD6XZGI4UJV7UCLUXAEQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX): protocol: expected piece hash; context canceled; EOF Oct 15 09:48:42.434666138 (524h17m27.896758621s) [noise=true , address=45.91.116.5:4299 , piece_id=4ARMXXE5LBRE2OZQEIITELKPTW4TISZWTD6XZGI4UJV7UCLUXAEQ, seg_pos={0 0}, upload=461218]: Long tail cancellation due to optimal threshold Oct 15 09:48:42.434692849 (524h17m27.896785432s) [noise=true , address=45.91.116.12:4226 , piece_id=DRXRGO7OXEKVS5ACL6FC7RKLGBBD6KNI7X6E5DM5D5VV5PEKN7FQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1): protocol: expected piece hash; context canceled; EOF Oct 15 09:48:42.434706184 (524h17m27.896798657s) [noise=true , address=45.91.116.12:4226 , piece_id=DRXRGO7OXEKVS5ACL6FC7RKLGBBD6KNI7X6E5DM5D5VV5PEKN7FQ, seg_pos={0 0}, upload=461218]: Long tail cancellation due to optimal threshold Oct 15 09:48:42.434718318 (524h17m27.896810911s) [noise=true , address=45.91.116.4:4940 , piece_id=3MCSUDYN73NIVJTGKPOVY6N3MALB3GBXI7I5NAX4HXPDHVMQ52BA, seg_pos={0 0}, upload=461218]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe): protocol: expected piece hash; context canceled; EOF Oct 15 09:48:42.434728447 (524h17m27.89682093s) [noise=true , address=45.91.116.4:4940 , piece_id=3MCSUDYN73NIVJTGKPOVY6N3MALB3GBXI7I5NAX4HXPDHVMQ52BA, seg_pos={0 0}, upload=461218]: Long tail cancellation due to optimal threshold Oct 15 09:48:42.434760458 (524h17m27.896852931s) [seg_pos={0 0}, upload=461218]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 15 09:48:42.434866900 (524h17m27.896959384s) [upload=461218]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 15 09:48:42.434873884 (524h17m27.896966367s) [upload=461218]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 15 09:48:42.760633614 (524h17m28.222726087s) [upload=461218]: Done uploading segments... Oct 15 09:48:42.760655435 (524h17m28.222747918s) [upload=461218]: Upload finished. err: <nil> Duration: 7.700587943s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...............................................X...X........] duration: 6.068331536s FAIL (2025-10-15 09:54:27 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, Piece ID: VH7ZJKSZ2NUXQJZE2IKNWP5PAL3TN2NNHD3ATZUQHGKKYMSSM3LQ) context deadline exceeded; piecestore: (Node ID: 1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N, Piece ID: UMQRQYPZ7SWUCO2N7JDCEVI5U5P36FDL3SUM6HIYJVWVOZTWNXQQ) context deadline exceeded ======LOGS====== Oct 15 09:48:38.801058027 (524h17m24.26315051s) [upload=461218]: Starting upload Oct 15 09:48:38.801138119 (524h17m24.263230592s) [upload=461218]: Done starting upload Oct 15 09:48:38.801288545 (524h17m24.263381028s) [upload=461218]: Uploading segments... Oct 15 09:48:38.801442509 (524h17m24.263535002s) [upload=461218]: Got next segment. Inline: false Oct 15 09:48:38.801486663 (524h17m24.263579136s) [upload=461218]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 15 09:48:38.801493245 (524h17m24.263585728s) [upload=461218]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 15 09:48:39.066131918 (524h17m24.528224401s) [seg_pos={0 0}, upload=461218]: Begin upload segment... Oct 15 09:48:39.066205549 (524h17m24.528298022s) [seg_pos={0 0}, upload=461218]: Done begin upload segment. Oct 15 09:48:39.066243090 (524h17m24.528335573s) [noise=true , address=45.91.116.5:4299 , piece_id=4ARMXXE5LBRE2OZQEIITELKPTW4TISZWTD6XZGI4UJV7UCLUXAEQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.066254431 (524h17m24.528346915s) [noise=true , address=45.91.116.11:4117 , piece_id=VH7ZJKSZ2NUXQJZE2IKNWP5PAL3TN2NNHD3ATZUQHGKKYMSSM3LQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.066874873 (524h17m24.528967356s) [noise=true , address=145.79.227.7:4294 , piece_id=H4ZNVIXORNO4MTXOVYDQZI4IJE4VXHZWC6JI35YKOR3UESSXJNTQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.067301676 (524h17m24.529394159s) [noise=true , address=45.91.116.9:4262 , piece_id=X53JZGRLZLX53UZLLCXXUQCFSQD2FO76X74OM45G26HNDU55UXXQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.067705735 (524h17m24.529798218s) [noise=true , address=145.79.227.4:4187 , piece_id=TQVPL6ZA6LHKAJVL6XWSYASZD3MCKKV3553CRFD72XWTVWY62STQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.069183369 (524h17m24.531275862s) [noise=true , address=145.79.227.5:4222 , piece_id=C2JSPN2QVSGSXG6ZW3ZMOFX5YFNN65XY7E62N5OEHOOMK3NSO4KQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.070534782 (524h17m24.532627265s) [noise=true , address=77.90.31.11:4555 , piece_id=U5RYZSD643KSUI4XLL26EAN6FQRPMG62GQMMCQRPEEG2XKSNBIZQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.072356982 (524h17m24.534449466s) [noise=true , address=145.79.227.11:4730 , piece_id=VO2DSM5LT43QAFPWOY3W3PVEJ44GZJWTCWG5QRQQFB363DFVFZDA, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.073961167 (524h17m24.53605364s) [noise=true , address=45.91.116.4:4940 , piece_id=3MCSUDYN73NIVJTGKPOVY6N3MALB3GBXI7I5NAX4HXPDHVMQ52BA, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.075716960 (524h17m24.537809433s) [noise=true , address=145.79.227.16:4910 , piece_id=BIHEJDH7RPI4UCPUNV453QHSEVCLA54PSNNQB2II37XZDRXFZNWQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.079266419 (524h17m24.541358902s) [noise=true , address=145.79.227.14:4838 , piece_id=6K6A4CNEHGB3X6DEBLF5KHLUT4GE6BQBW6IE3RHBML2NWPN255AA, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.080943032 (524h17m24.543035505s) [noise=true , address=45.91.116.12:4226 , piece_id=DRXRGO7OXEKVS5ACL6FC7RKLGBBD6KNI7X6E5DM5D5VV5PEKN7FQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.081340479 (524h17m24.543432962s) [noise=true , address=45.91.116.8:4045 , piece_id=EDXBNLKKRZMT6MKK2KOIZM7DDKA3IHB6UMCXX4YKZF7JY2T3S5CQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.083616223 (524h17m24.545708706s) [noise=true , address=145.79.227.14:4840 , piece_id=B4ZVPRMVH7Y3TCWCLPO7DWRVMJL7VSI73CIT4PEHCMTIEURQKB2Q, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.083633946 (524h17m24.545726429s) [noise=true , address=45.91.116.14:4189 , piece_id=UMQRQYPZ7SWUCO2N7JDCEVI5U5P36FDL3SUM6HIYJVWVOZTWNXQQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.085493247 (524h17m24.54758573s) [noise=true , address=145.79.227.17:4969 , piece_id=R5HIP33QW5HLICARFWDUOW7I75YTDHZTJCIML5X52VKHLPL3J4YQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:40.727263388 (524h17m26.189355871s) [upload=461218]: Next returned nil segment Oct 15 09:48:40.727310076 (524h17m26.189402549s) [upload=461218]: Waiting for error group managing segments... Oct 15 09:48:41.004514306 (524h17m26.466606789s) [noise=true , address=145.79.227.7:4294 , piece_id=H4ZNVIXORNO4MTXOVYDQZI4IJE4VXHZWC6JI35YKOR3UESSXJNTQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.006116557 (524h17m26.46820904s) [noise=true , address=145.79.227.5:4222 , piece_id=C2JSPN2QVSGSXG6ZW3ZMOFX5YFNN65XY7E62N5OEHOOMK3NSO4KQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.012963184 (524h17m26.475055667s) [noise=true , address=145.79.227.16:4910 , piece_id=BIHEJDH7RPI4UCPUNV453QHSEVCLA54PSNNQB2II37XZDRXFZNWQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.012994684 (524h17m26.475087167s) [noise=true , address=145.79.227.14:4840 , piece_id=B4ZVPRMVH7Y3TCWCLPO7DWRVMJL7VSI73CIT4PEHCMTIEURQKB2Q, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.016649956 (524h17m26.478742439s) [noise=true , address=145.79.227.11:4730 , piece_id=VO2DSM5LT43QAFPWOY3W3PVEJ44GZJWTCWG5QRQQFB363DFVFZDA, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.019248864 (524h17m26.481341347s) [noise=true , address=145.79.227.17:4969 , piece_id=R5HIP33QW5HLICARFWDUOW7I75YTDHZTJCIML5X52VKHLPL3J4YQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.075991585 (524h17m26.538084068s) [noise=true , address=145.79.227.4:4187 , piece_id=TQVPL6ZA6LHKAJVL6XWSYASZD3MCKKV3553CRFD72XWTVWY62STQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.118655146 (524h17m26.580747619s) [noise=true , address=145.79.227.14:4838 , piece_id=6K6A4CNEHGB3X6DEBLF5KHLUT4GE6BQBW6IE3RHBML2NWPN255AA, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.285934795 (524h17m26.748027278s) [noise=true , address=77.90.31.11:4555 , piece_id=U5RYZSD643KSUI4XLL26EAN6FQRPMG62GQMMCQRPEEG2XKSNBIZQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.744525465 (524h17m27.206617948s) [noise=true , address=45.91.116.9:4262 , piece_id=X53JZGRLZLX53UZLLCXXUQCFSQD2FO76X74OM45G26HNDU55UXXQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:42.218961989 (524h17m27.681054462s) [noise=true , address=45.91.116.11:4117 , piece_id=VH7ZJKSZ2NUXQJZE2IKNWP5PAL3TN2NNHD3ATZUQHGKKYMSSM3LQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:42.313638675 (524h17m27.775731158s) [noise=true , address=45.91.116.14:4189 , piece_id=UMQRQYPZ7SWUCO2N7JDCEVI5U5P36FDL3SUM6HIYJVWVOZTWNXQQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:42.434283559 (524h17m27.896376053s) [noise=true , address=45.91.116.8:4045 , piece_id=EDXBNLKKRZMT6MKK2KOIZM7DDKA3IHB6UMCXX4YKZF7JY2T3S5CQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:42.434321832 (524h17m27.896414305s) [seg_pos={0 0}, upload=461218]: Segment reached optimal threshold of 13 pieces. Oct 15 09:48:42.434605041 (524h17m27.896697534s) [noise=true , address=45.91.116.5:4299 , piece_id=4ARMXXE5LBRE2OZQEIITELKPTW4TISZWTD6XZGI4UJV7UCLUXAEQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX): protocol: expected piece hash; context canceled; EOF Oct 15 09:48:42.434666138 (524h17m27.896758621s) [noise=true , address=45.91.116.5:4299 , piece_id=4ARMXXE5LBRE2OZQEIITELKPTW4TISZWTD6XZGI4UJV7UCLUXAEQ, seg_pos={0 0}, upload=461218]: Long tail cancellation due to optimal threshold Oct 15 09:48:42.434692849 (524h17m27.896785432s) [noise=true , address=45.91.116.12:4226 , piece_id=DRXRGO7OXEKVS5ACL6FC7RKLGBBD6KNI7X6E5DM5D5VV5PEKN7FQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1): protocol: expected piece hash; context canceled; EOF Oct 15 09:48:42.434706184 (524h17m27.896798657s) [noise=true , address=45.91.116.12:4226 , piece_id=DRXRGO7OXEKVS5ACL6FC7RKLGBBD6KNI7X6E5DM5D5VV5PEKN7FQ, seg_pos={0 0}, upload=461218]: Long tail cancellation due to optimal threshold Oct 15 09:48:42.434718318 (524h17m27.896810911s) [noise=true , address=45.91.116.4:4940 , piece_id=3MCSUDYN73NIVJTGKPOVY6N3MALB3GBXI7I5NAX4HXPDHVMQ52BA, seg_pos={0 0}, upload=461218]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe): protocol: expected piece hash; context canceled; EOF Oct 15 09:48:42.434728447 (524h17m27.89682093s) [noise=true , address=45.91.116.4:4940 , piece_id=3MCSUDYN73NIVJTGKPOVY6N3MALB3GBXI7I5NAX4HXPDHVMQ52BA, seg_pos={0 0}, upload=461218]: Long tail cancellation due to optimal threshold Oct 15 09:48:42.434760458 (524h17m27.896852931s) [seg_pos={0 0}, upload=461218]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 15 09:48:42.434866900 (524h17m27.896959384s) [upload=461218]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 15 09:48:42.434873884 (524h17m27.896966367s) [upload=461218]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 15 09:48:42.760633614 (524h17m28.222726087s) [upload=461218]: Done uploading segments... Oct 15 09:48:42.760655435 (524h17m28.222747918s) [upload=461218]: Upload finished. err: <nil> Duration: 6.068391871s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [................................................X...X.......] duration: 6.514897054s FAIL (2025-10-15 09:53:34 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, Piece ID: VH7ZJKSZ2NUXQJZE2IKNWP5PAL3TN2NNHD3ATZUQHGKKYMSSM3LQ) context deadline exceeded; piecestore: (Node ID: 1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N, Piece ID: UMQRQYPZ7SWUCO2N7JDCEVI5U5P36FDL3SUM6HIYJVWVOZTWNXQQ) context deadline exceeded ======LOGS====== Oct 15 09:48:38.801058027 (524h17m24.26315051s) [upload=461218]: Starting upload Oct 15 09:48:38.801138119 (524h17m24.263230592s) [upload=461218]: Done starting upload Oct 15 09:48:38.801288545 (524h17m24.263381028s) [upload=461218]: Uploading segments... Oct 15 09:48:38.801442509 (524h17m24.263535002s) [upload=461218]: Got next segment. Inline: false Oct 15 09:48:38.801486663 (524h17m24.263579136s) [upload=461218]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 15 09:48:38.801493245 (524h17m24.263585728s) [upload=461218]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 15 09:48:39.066131918 (524h17m24.528224401s) [seg_pos={0 0}, upload=461218]: Begin upload segment... Oct 15 09:48:39.066205549 (524h17m24.528298022s) [seg_pos={0 0}, upload=461218]: Done begin upload segment. Oct 15 09:48:39.066243090 (524h17m24.528335573s) [noise=true , address=45.91.116.5:4299 , piece_id=4ARMXXE5LBRE2OZQEIITELKPTW4TISZWTD6XZGI4UJV7UCLUXAEQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.066254431 (524h17m24.528346915s) [noise=true , address=45.91.116.11:4117 , piece_id=VH7ZJKSZ2NUXQJZE2IKNWP5PAL3TN2NNHD3ATZUQHGKKYMSSM3LQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.066874873 (524h17m24.528967356s) [noise=true , address=145.79.227.7:4294 , piece_id=H4ZNVIXORNO4MTXOVYDQZI4IJE4VXHZWC6JI35YKOR3UESSXJNTQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.067301676 (524h17m24.529394159s) [noise=true , address=45.91.116.9:4262 , piece_id=X53JZGRLZLX53UZLLCXXUQCFSQD2FO76X74OM45G26HNDU55UXXQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.067705735 (524h17m24.529798218s) [noise=true , address=145.79.227.4:4187 , piece_id=TQVPL6ZA6LHKAJVL6XWSYASZD3MCKKV3553CRFD72XWTVWY62STQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.069183369 (524h17m24.531275862s) [noise=true , address=145.79.227.5:4222 , piece_id=C2JSPN2QVSGSXG6ZW3ZMOFX5YFNN65XY7E62N5OEHOOMK3NSO4KQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.070534782 (524h17m24.532627265s) [noise=true , address=77.90.31.11:4555 , piece_id=U5RYZSD643KSUI4XLL26EAN6FQRPMG62GQMMCQRPEEG2XKSNBIZQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.072356982 (524h17m24.534449466s) [noise=true , address=145.79.227.11:4730 , piece_id=VO2DSM5LT43QAFPWOY3W3PVEJ44GZJWTCWG5QRQQFB363DFVFZDA, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.073961167 (524h17m24.53605364s) [noise=true , address=45.91.116.4:4940 , piece_id=3MCSUDYN73NIVJTGKPOVY6N3MALB3GBXI7I5NAX4HXPDHVMQ52BA, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.075716960 (524h17m24.537809433s) [noise=true , address=145.79.227.16:4910 , piece_id=BIHEJDH7RPI4UCPUNV453QHSEVCLA54PSNNQB2II37XZDRXFZNWQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.079266419 (524h17m24.541358902s) [noise=true , address=145.79.227.14:4838 , piece_id=6K6A4CNEHGB3X6DEBLF5KHLUT4GE6BQBW6IE3RHBML2NWPN255AA, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.080943032 (524h17m24.543035505s) [noise=true , address=45.91.116.12:4226 , piece_id=DRXRGO7OXEKVS5ACL6FC7RKLGBBD6KNI7X6E5DM5D5VV5PEKN7FQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.081340479 (524h17m24.543432962s) [noise=true , address=45.91.116.8:4045 , piece_id=EDXBNLKKRZMT6MKK2KOIZM7DDKA3IHB6UMCXX4YKZF7JY2T3S5CQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.083616223 (524h17m24.545708706s) [noise=true , address=145.79.227.14:4840 , piece_id=B4ZVPRMVH7Y3TCWCLPO7DWRVMJL7VSI73CIT4PEHCMTIEURQKB2Q, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.083633946 (524h17m24.545726429s) [noise=true , address=45.91.116.14:4189 , piece_id=UMQRQYPZ7SWUCO2N7JDCEVI5U5P36FDL3SUM6HIYJVWVOZTWNXQQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.085493247 (524h17m24.54758573s) [noise=true , address=145.79.227.17:4969 , piece_id=R5HIP33QW5HLICARFWDUOW7I75YTDHZTJCIML5X52VKHLPL3J4YQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:40.727263388 (524h17m26.189355871s) [upload=461218]: Next returned nil segment Oct 15 09:48:40.727310076 (524h17m26.189402549s) [upload=461218]: Waiting for error group managing segments... Oct 15 09:48:41.004514306 (524h17m26.466606789s) [noise=true , address=145.79.227.7:4294 , piece_id=H4ZNVIXORNO4MTXOVYDQZI4IJE4VXHZWC6JI35YKOR3UESSXJNTQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.006116557 (524h17m26.46820904s) [noise=true , address=145.79.227.5:4222 , piece_id=C2JSPN2QVSGSXG6ZW3ZMOFX5YFNN65XY7E62N5OEHOOMK3NSO4KQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.012963184 (524h17m26.475055667s) [noise=true , address=145.79.227.16:4910 , piece_id=BIHEJDH7RPI4UCPUNV453QHSEVCLA54PSNNQB2II37XZDRXFZNWQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.012994684 (524h17m26.475087167s) [noise=true , address=145.79.227.14:4840 , piece_id=B4ZVPRMVH7Y3TCWCLPO7DWRVMJL7VSI73CIT4PEHCMTIEURQKB2Q, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.016649956 (524h17m26.478742439s) [noise=true , address=145.79.227.11:4730 , piece_id=VO2DSM5LT43QAFPWOY3W3PVEJ44GZJWTCWG5QRQQFB363DFVFZDA, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.019248864 (524h17m26.481341347s) [noise=true , address=145.79.227.17:4969 , piece_id=R5HIP33QW5HLICARFWDUOW7I75YTDHZTJCIML5X52VKHLPL3J4YQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.075991585 (524h17m26.538084068s) [noise=true , address=145.79.227.4:4187 , piece_id=TQVPL6ZA6LHKAJVL6XWSYASZD3MCKKV3553CRFD72XWTVWY62STQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.118655146 (524h17m26.580747619s) [noise=true , address=145.79.227.14:4838 , piece_id=6K6A4CNEHGB3X6DEBLF5KHLUT4GE6BQBW6IE3RHBML2NWPN255AA, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.285934795 (524h17m26.748027278s) [noise=true , address=77.90.31.11:4555 , piece_id=U5RYZSD643KSUI4XLL26EAN6FQRPMG62GQMMCQRPEEG2XKSNBIZQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.744525465 (524h17m27.206617948s) [noise=true , address=45.91.116.9:4262 , piece_id=X53JZGRLZLX53UZLLCXXUQCFSQD2FO76X74OM45G26HNDU55UXXQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:42.218961989 (524h17m27.681054462s) [noise=true , address=45.91.116.11:4117 , piece_id=VH7ZJKSZ2NUXQJZE2IKNWP5PAL3TN2NNHD3ATZUQHGKKYMSSM3LQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:42.313638675 (524h17m27.775731158s) [noise=true , address=45.91.116.14:4189 , piece_id=UMQRQYPZ7SWUCO2N7JDCEVI5U5P36FDL3SUM6HIYJVWVOZTWNXQQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:42.434283559 (524h17m27.896376053s) [noise=true , address=45.91.116.8:4045 , piece_id=EDXBNLKKRZMT6MKK2KOIZM7DDKA3IHB6UMCXX4YKZF7JY2T3S5CQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:42.434321832 (524h17m27.896414305s) [seg_pos={0 0}, upload=461218]: Segment reached optimal threshold of 13 pieces. Oct 15 09:48:42.434605041 (524h17m27.896697534s) [noise=true , address=45.91.116.5:4299 , piece_id=4ARMXXE5LBRE2OZQEIITELKPTW4TISZWTD6XZGI4UJV7UCLUXAEQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX): protocol: expected piece hash; context canceled; EOF Oct 15 09:48:42.434666138 (524h17m27.896758621s) [noise=true , address=45.91.116.5:4299 , piece_id=4ARMXXE5LBRE2OZQEIITELKPTW4TISZWTD6XZGI4UJV7UCLUXAEQ, seg_pos={0 0}, upload=461218]: Long tail cancellation due to optimal threshold Oct 15 09:48:42.434692849 (524h17m27.896785432s) [noise=true , address=45.91.116.12:4226 , piece_id=DRXRGO7OXEKVS5ACL6FC7RKLGBBD6KNI7X6E5DM5D5VV5PEKN7FQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1): protocol: expected piece hash; context canceled; EOF Oct 15 09:48:42.434706184 (524h17m27.896798657s) [noise=true , address=45.91.116.12:4226 , piece_id=DRXRGO7OXEKVS5ACL6FC7RKLGBBD6KNI7X6E5DM5D5VV5PEKN7FQ, seg_pos={0 0}, upload=461218]: Long tail cancellation due to optimal threshold Oct 15 09:48:42.434718318 (524h17m27.896810911s) [noise=true , address=45.91.116.4:4940 , piece_id=3MCSUDYN73NIVJTGKPOVY6N3MALB3GBXI7I5NAX4HXPDHVMQ52BA, seg_pos={0 0}, upload=461218]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe): protocol: expected piece hash; context canceled; EOF Oct 15 09:48:42.434728447 (524h17m27.89682093s) [noise=true , address=45.91.116.4:4940 , piece_id=3MCSUDYN73NIVJTGKPOVY6N3MALB3GBXI7I5NAX4HXPDHVMQ52BA, seg_pos={0 0}, upload=461218]: Long tail cancellation due to optimal threshold Oct 15 09:48:42.434760458 (524h17m27.896852931s) [seg_pos={0 0}, upload=461218]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 15 09:48:42.434866900 (524h17m27.896959384s) [upload=461218]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 15 09:48:42.434873884 (524h17m27.896966367s) [upload=461218]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 15 09:48:42.760633614 (524h17m28.222726087s) [upload=461218]: Done uploading segments... Oct 15 09:48:42.760655435 (524h17m28.222747918s) [upload=461218]: Upload finished. err: <nil> Duration: 6.514933242s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.................................................X...X......] duration: 6.41787935s FAIL (2025-10-15 09:53:08 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, Piece ID: VH7ZJKSZ2NUXQJZE2IKNWP5PAL3TN2NNHD3ATZUQHGKKYMSSM3LQ) context deadline exceeded; piecestore: (Node ID: 1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N, Piece ID: UMQRQYPZ7SWUCO2N7JDCEVI5U5P36FDL3SUM6HIYJVWVOZTWNXQQ) context deadline exceeded ======LOGS====== Oct 15 09:48:38.801058027 (524h17m24.26315051s) [upload=461218]: Starting upload Oct 15 09:48:38.801138119 (524h17m24.263230592s) [upload=461218]: Done starting upload Oct 15 09:48:38.801288545 (524h17m24.263381028s) [upload=461218]: Uploading segments... Oct 15 09:48:38.801442509 (524h17m24.263535002s) [upload=461218]: Got next segment. Inline: false Oct 15 09:48:38.801486663 (524h17m24.263579136s) [upload=461218]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 15 09:48:38.801493245 (524h17m24.263585728s) [upload=461218]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 15 09:48:39.066131918 (524h17m24.528224401s) [seg_pos={0 0}, upload=461218]: Begin upload segment... Oct 15 09:48:39.066205549 (524h17m24.528298022s) [seg_pos={0 0}, upload=461218]: Done begin upload segment. Oct 15 09:48:39.066243090 (524h17m24.528335573s) [noise=true , address=45.91.116.5:4299 , piece_id=4ARMXXE5LBRE2OZQEIITELKPTW4TISZWTD6XZGI4UJV7UCLUXAEQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.066254431 (524h17m24.528346915s) [noise=true , address=45.91.116.11:4117 , piece_id=VH7ZJKSZ2NUXQJZE2IKNWP5PAL3TN2NNHD3ATZUQHGKKYMSSM3LQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.066874873 (524h17m24.528967356s) [noise=true , address=145.79.227.7:4294 , piece_id=H4ZNVIXORNO4MTXOVYDQZI4IJE4VXHZWC6JI35YKOR3UESSXJNTQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.067301676 (524h17m24.529394159s) [noise=true , address=45.91.116.9:4262 , piece_id=X53JZGRLZLX53UZLLCXXUQCFSQD2FO76X74OM45G26HNDU55UXXQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.067705735 (524h17m24.529798218s) [noise=true , address=145.79.227.4:4187 , piece_id=TQVPL6ZA6LHKAJVL6XWSYASZD3MCKKV3553CRFD72XWTVWY62STQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.069183369 (524h17m24.531275862s) [noise=true , address=145.79.227.5:4222 , piece_id=C2JSPN2QVSGSXG6ZW3ZMOFX5YFNN65XY7E62N5OEHOOMK3NSO4KQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.070534782 (524h17m24.532627265s) [noise=true , address=77.90.31.11:4555 , piece_id=U5RYZSD643KSUI4XLL26EAN6FQRPMG62GQMMCQRPEEG2XKSNBIZQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.072356982 (524h17m24.534449466s) [noise=true , address=145.79.227.11:4730 , piece_id=VO2DSM5LT43QAFPWOY3W3PVEJ44GZJWTCWG5QRQQFB363DFVFZDA, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.073961167 (524h17m24.53605364s) [noise=true , address=45.91.116.4:4940 , piece_id=3MCSUDYN73NIVJTGKPOVY6N3MALB3GBXI7I5NAX4HXPDHVMQ52BA, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.075716960 (524h17m24.537809433s) [noise=true , address=145.79.227.16:4910 , piece_id=BIHEJDH7RPI4UCPUNV453QHSEVCLA54PSNNQB2II37XZDRXFZNWQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.079266419 (524h17m24.541358902s) [noise=true , address=145.79.227.14:4838 , piece_id=6K6A4CNEHGB3X6DEBLF5KHLUT4GE6BQBW6IE3RHBML2NWPN255AA, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.080943032 (524h17m24.543035505s) [noise=true , address=45.91.116.12:4226 , piece_id=DRXRGO7OXEKVS5ACL6FC7RKLGBBD6KNI7X6E5DM5D5VV5PEKN7FQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.081340479 (524h17m24.543432962s) [noise=true , address=45.91.116.8:4045 , piece_id=EDXBNLKKRZMT6MKK2KOIZM7DDKA3IHB6UMCXX4YKZF7JY2T3S5CQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.083616223 (524h17m24.545708706s) [noise=true , address=145.79.227.14:4840 , piece_id=B4ZVPRMVH7Y3TCWCLPO7DWRVMJL7VSI73CIT4PEHCMTIEURQKB2Q, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.083633946 (524h17m24.545726429s) [noise=true , address=45.91.116.14:4189 , piece_id=UMQRQYPZ7SWUCO2N7JDCEVI5U5P36FDL3SUM6HIYJVWVOZTWNXQQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:39.085493247 (524h17m24.54758573s) [noise=true , address=145.79.227.17:4969 , piece_id=R5HIP33QW5HLICARFWDUOW7I75YTDHZTJCIML5X52VKHLPL3J4YQ, seg_pos={0 0}, upload=461218]: Uploading piece... Oct 15 09:48:40.727263388 (524h17m26.189355871s) [upload=461218]: Next returned nil segment Oct 15 09:48:40.727310076 (524h17m26.189402549s) [upload=461218]: Waiting for error group managing segments... Oct 15 09:48:41.004514306 (524h17m26.466606789s) [noise=true , address=145.79.227.7:4294 , piece_id=H4ZNVIXORNO4MTXOVYDQZI4IJE4VXHZWC6JI35YKOR3UESSXJNTQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.006116557 (524h17m26.46820904s) [noise=true , address=145.79.227.5:4222 , piece_id=C2JSPN2QVSGSXG6ZW3ZMOFX5YFNN65XY7E62N5OEHOOMK3NSO4KQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.012963184 (524h17m26.475055667s) [noise=true , address=145.79.227.16:4910 , piece_id=BIHEJDH7RPI4UCPUNV453QHSEVCLA54PSNNQB2II37XZDRXFZNWQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.012994684 (524h17m26.475087167s) [noise=true , address=145.79.227.14:4840 , piece_id=B4ZVPRMVH7Y3TCWCLPO7DWRVMJL7VSI73CIT4PEHCMTIEURQKB2Q, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.016649956 (524h17m26.478742439s) [noise=true , address=145.79.227.11:4730 , piece_id=VO2DSM5LT43QAFPWOY3W3PVEJ44GZJWTCWG5QRQQFB363DFVFZDA, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.019248864 (524h17m26.481341347s) [noise=true , address=145.79.227.17:4969 , piece_id=R5HIP33QW5HLICARFWDUOW7I75YTDHZTJCIML5X52VKHLPL3J4YQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.075991585 (524h17m26.538084068s) [noise=true , address=145.79.227.4:4187 , piece_id=TQVPL6ZA6LHKAJVL6XWSYASZD3MCKKV3553CRFD72XWTVWY62STQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.118655146 (524h17m26.580747619s) [noise=true , address=145.79.227.14:4838 , piece_id=6K6A4CNEHGB3X6DEBLF5KHLUT4GE6BQBW6IE3RHBML2NWPN255AA, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.285934795 (524h17m26.748027278s) [noise=true , address=77.90.31.11:4555 , piece_id=U5RYZSD643KSUI4XLL26EAN6FQRPMG62GQMMCQRPEEG2XKSNBIZQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:41.744525465 (524h17m27.206617948s) [noise=true , address=45.91.116.9:4262 , piece_id=X53JZGRLZLX53UZLLCXXUQCFSQD2FO76X74OM45G26HNDU55UXXQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:42.218961989 (524h17m27.681054462s) [noise=true , address=45.91.116.11:4117 , piece_id=VH7ZJKSZ2NUXQJZE2IKNWP5PAL3TN2NNHD3ATZUQHGKKYMSSM3LQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:42.313638675 (524h17m27.775731158s) [noise=true , address=45.91.116.14:4189 , piece_id=UMQRQYPZ7SWUCO2N7JDCEVI5U5P36FDL3SUM6HIYJVWVOZTWNXQQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:42.434283559 (524h17m27.896376053s) [noise=true , address=45.91.116.8:4045 , piece_id=EDXBNLKKRZMT6MKK2KOIZM7DDKA3IHB6UMCXX4YKZF7JY2T3S5CQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: <nil> Oct 15 09:48:42.434321832 (524h17m27.896414305s) [seg_pos={0 0}, upload=461218]: Segment reached optimal threshold of 13 pieces. Oct 15 09:48:42.434605041 (524h17m27.896697534s) [noise=true , address=45.91.116.5:4299 , piece_id=4ARMXXE5LBRE2OZQEIITELKPTW4TISZWTD6XZGI4UJV7UCLUXAEQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX): protocol: expected piece hash; context canceled; EOF Oct 15 09:48:42.434666138 (524h17m27.896758621s) [noise=true , address=45.91.116.5:4299 , piece_id=4ARMXXE5LBRE2OZQEIITELKPTW4TISZWTD6XZGI4UJV7UCLUXAEQ, seg_pos={0 0}, upload=461218]: Long tail cancellation due to optimal threshold Oct 15 09:48:42.434692849 (524h17m27.896785432s) [noise=true , address=45.91.116.12:4226 , piece_id=DRXRGO7OXEKVS5ACL6FC7RKLGBBD6KNI7X6E5DM5D5VV5PEKN7FQ, seg_pos={0 0}, upload=461218]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1): protocol: expected piece hash; context canceled; EOF Oct 15 09:48:42.434706184 (524h17m27.896798657s) [noise=true , address=45.91.116.12:4226 , piece_id=DRXRGO7OXEKVS5ACL6FC7RKLGBBD6KNI7X6E5DM5D5VV5PEKN7FQ, seg_pos={0 0}, upload=461218]: Long tail cancellation due to optimal threshold Oct 15 09:48:42.434718318 (524h17m27.896810911s) [noise=true , address=45.91.116.4:4940 , piece_id=3MCSUDYN73NIVJTGKPOVY6N3MALB3GBXI7I5NAX4HXPDHVMQ52BA, seg_pos={0 0}, upload=461218]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe): protocol: expected piece hash; context canceled; EOF Oct 15 09:48:42.434728447 (524h17m27.89682093s) [noise=true , address=45.91.116.4:4940 , piece_id=3MCSUDYN73NIVJTGKPOVY6N3MALB3GBXI7I5NAX4HXPDHVMQ52BA, seg_pos={0 0}, upload=461218]: Long tail cancellation due to optimal threshold Oct 15 09:48:42.434760458 (524h17m27.896852931s) [seg_pos={0 0}, upload=461218]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 15 09:48:42.434866900 (524h17m27.896959384s) [upload=461218]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 15 09:48:42.434873884 (524h17m27.896966367s) [upload=461218]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 15 09:48:42.760633614 (524h17m28.222726087s) [upload=461218]: Done uploading segments... Oct 15 09:48:42.760655435 (524h17m28.222747918s) [upload=461218]: Upload finished. err: <nil> Duration: 6.417993306s core uplink us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)