Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..............X............................................X] duration: 30.231678668s FAIL (2025-08-09 12:12:10 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 3) ======LOGS====== 45.91.116.5:4299 , piece_id=UIWKRBUGTF6VLKL44CRFBHMTC3CUEHCWMFKECTRB33GRIOWEDMZQ, seg_pos={0 0}, upload=19598]: Uploading piece... Aug 9 12:11:41.574674700 (18h45m36.005754831s) [noise=true , address=45.91.116.14:4190 , piece_id=5XCAGAJOMW6XMWRN5LV5YVXR5UG6456Q6G6QEGDRKQJV66TJQUHQ, seg_pos={0 0}, upload=19598]: Uploading piece... Aug 9 12:11:41.574769040 (18h45m36.005849161s) [noise=true , address=45.91.116.10:4083 , piece_id=HGTE3QZ5IK3ZKGJQEQBRN7R6QT3YLMNZRTJ4GEVMFD3GJVILBM7A, seg_pos={0 0}, upload=19598]: Uploading piece... Aug 9 12:11:41.574996094 (18h45m36.006076225s) [noise=true , address=45.91.116.12:4227 , piece_id=RGAP7CQ523KMCMZS4RYHRU2LUXV4GOBZNBPGVXGENUWP4JVNOKAA, seg_pos={0 0}, upload=19598]: Uploading piece... Aug 9 12:11:43.368914296 (18h45m37.799994427s) [upload=19598]: Next returned nil segment Aug 9 12:11:43.368930026 (18h45m37.800010157s) [upload=19598]: Waiting for error group managing segments... Aug 9 12:11:43.660196363 (18h45m38.091276484s) [noise=true , address=104.36.82.12:4150 , piece_id=JNWZ7S7AO5SQFFTOHVH7IKDC5JAXYAKPKBXCWROVWJ3YSO35D7DQ, seg_pos={0 0}, upload=19598]: Done uploading piece. err: <nil> Aug 9 12:11:43.680836564 (18h45m38.111916695s) [noise=true , address=104.36.82.13:4185 , piece_id=MBHNLNBHCDBNQQWUQVZU2MP4MSFEUWV462HPBLBI2TNR3ADVV4CQ, seg_pos={0 0}, upload=19598]: Done uploading piece. err: <nil> Aug 9 12:11:43.752377904 (18h45m38.183458035s) [noise=true , address=104.36.82.7:4912 , piece_id=QOJEGBIYNAMJJNMBMJDZV74IM2TNWEY4KYRHKFJFQ7X7ITBNIB3A, seg_pos={0 0}, upload=19598]: Done uploading piece. err: <nil> Aug 9 12:12:10.687712282 (18h46m5.118792413s) [noise=true , address=45.91.116.8:4047 , piece_id=E6DUYWMGXKSEMOTWEZKVSPC73D542UQU3Q6HLWVACS46ZKZR24KQ, seg_pos={0 0}, upload=19598]: Done uploading piece. err: ecclient: upload failed (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN, address:45.91.116.8:4047): protocol: write tcp 188.166.225.170:49590->45.91.116.8:4047: use of closed network connection; write tcp 188.166.225.170:49590->45.91.116.8:4047: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49590->45.91.116.8:4047: use of closed network connection Aug 9 12:12:10.687789359 (18h46m5.11886948s) [noise=true , address=77.90.31.12:4592 , piece_id=ZEBVYQMZACTLEEUMQ53JUYZBCRE47BO4LVF2NNSWDAJ74TW3425Q, seg_pos={0 0}, upload=19598]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: write tcp 188.166.225.170:60610->77.90.31.12:4592: use of closed network connection; write tcp 188.166.225.170:60610->77.90.31.12:4592: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60610->77.90.31.12:4592: use of closed network connection Aug 9 12:12:10.687829085 (18h46m5.118909206s) [noise=true , address=77.90.31.16:4734 , piece_id=ZTVFF3NCR3NDVHSNZTPYBKUV4IFCBB4HGFJJCJ2XW2NFP674IBUA, seg_pos={0 0}, upload=19598]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: write tcp 188.166.225.170:49622->77.90.31.16:4734: use of closed network connection; write tcp 188.166.225.170:49622->77.90.31.16:4734: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49622->77.90.31.16:4734: use of closed network connection Aug 9 12:12:10.687878540 (18h46m5.118958671s) [noise=true , address=45.91.116.5:4299 , piece_id=UIWKRBUGTF6VLKL44CRFBHMTC3CUEHCWMFKECTRB33GRIOWEDMZQ, seg_pos={0 0}, upload=19598]: Done uploading piece. err: ecclient: upload failed (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX, address:45.91.116.5:4299): protocol: write tcp 188.166.225.170:55182->45.91.116.5:4299: use of closed network connection; write tcp 188.166.225.170:55182->45.91.116.5:4299: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55182->45.91.116.5:4299: use of closed network connection Aug 9 12:12:10.687909589 (18h46m5.11898972s) [noise=true , address=45.91.116.10:4083 , piece_id=HGTE3QZ5IK3ZKGJQEQBRN7R6QT3YLMNZRTJ4GEVMFD3GJVILBM7A, seg_pos={0 0}, upload=19598]: Done uploading piece. err: ecclient: upload failed (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B, address:45.91.116.10:4083): protocol: write tcp 188.166.225.170:45388->45.91.116.10:4083: use of closed network connection; write tcp 188.166.225.170:45388->45.91.116.10:4083: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45388->45.91.116.10:4083: use of closed network connection Aug 9 12:12:10.687935358 (18h46m5.119015479s) [noise=true , address=45.91.116.12:4227 , piece_id=RGAP7CQ523KMCMZS4RYHRU2LUXV4GOBZNBPGVXGENUWP4JVNOKAA, seg_pos={0 0}, upload=19598]: Done uploading piece. err: ecclient: upload failed (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, address:45.91.116.12:4227): protocol: write tcp 188.166.225.170:42694->45.91.116.12:4227: use of closed network connection; write tcp 188.166.225.170:42694->45.91.116.12:4227: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42694->45.91.116.12:4227: use of closed network connection Aug 9 12:12:10.687990334 (18h46m5.119070465s) [noise=true , address=45.91.116.7:4010 , piece_id=FJUK7Z62XPW4VVJZVYH4EMUQAF63QG5VSCHRWBVDTQ6NYM4VY5UQ, seg_pos={0 0}, upload=19598]: Done uploading piece. err: ecclient: upload failed (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w, address:45.91.116.7:4010): protocol: write tcp 188.166.225.170:46086->45.91.116.7:4010: use of closed network connection; write tcp 188.166.225.170:46086->45.91.116.7:4010: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46086->45.91.116.7:4010: use of closed network connection Aug 9 12:12:10.688056110 (18h46m5.119136241s) [noise=true , address=45.91.116.9:4263 , piece_id=2PZSPO4KCUM3Q37HF4NMCSJHCYBMJG4NTXYTMEX3RZKKHU27F7CA, seg_pos={0 0}, upload=19598]: Done uploading piece. err: ecclient: upload failed (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF, address:45.91.116.9:4263): protocol: write tcp 188.166.225.170:49230->45.91.116.9:4263: use of closed network connection; write tcp 188.166.225.170:49230->45.91.116.9:4263: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49230->45.91.116.9:4263: use of closed network connection Aug 9 12:12:10.688125162 (18h46m5.119205283s) [noise=true , address=45.91.116.4:4939 , piece_id=UD6XGDJ562QJSVDHJMQPSSAW5GRMD47N3KCWKOGJYWWEXPJBNIFA, seg_pos={0 0}, upload=19598]: Done uploading piece. err: ecclient: upload failed (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX, address:45.91.116.4:4939): protocol: write tcp 188.166.225.170:37202->45.91.116.4:4939: use of closed network connection; write tcp 188.166.225.170:37202->45.91.116.4:4939: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37202->45.91.116.4:4939: use of closed network connection Aug 9 12:12:10.688142255 (18h46m5.119222386s) [noise=true , address=77.90.31.17:4770 , piece_id=POJ7KJ6RIOSOPGLW35SF3HV6DXM3V2Q3GBSF22UPE32IT6S2ZRSA, seg_pos={0 0}, upload=19598]: Done uploading piece. err: ecclient: upload failed (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R, address:77.90.31.17:4770): protocol: write tcp 188.166.225.170:53758->77.90.31.17:4770: use of closed network connection; write tcp 188.166.225.170:53758->77.90.31.17:4770: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53758->77.90.31.17:4770: use of closed network connection Aug 9 12:12:10.688215314 (18h46m5.119295435s) [noise=true , address=45.91.116.14:4190 , piece_id=5XCAGAJOMW6XMWRN5LV5YVXR5UG6456Q6G6QEGDRKQJV66TJQUHQ, seg_pos={0 0}, upload=19598]: Done uploading piece. err: ecclient: upload failed (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, address:45.91.116.14:4190): protocol: write tcp 188.166.225.170:36620->45.91.116.14:4190: use of closed network connection; write tcp 188.166.225.170:36620->45.91.116.14:4190: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36620->45.91.116.14:4190: use of closed network connection Aug 9 12:12:10.688253928 (18h46m5.119334059s) [noise=true , address=77.90.31.11:4554 , piece_id=OD352ZZCM5HO5PCCVQDYMBSJVOI4OMA4TNYQIORMCIRZOGSSHKJA, seg_pos={0 0}, upload=19598]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: write tcp 188.166.225.170:54534->77.90.31.11:4554: use of closed network connection; write tcp 188.166.225.170:54534->77.90.31.11:4554: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54534->77.90.31.11:4554: use of closed network connection Aug 9 12:12:10.688275490 (18h46m5.119355621s) [noise=true , address=45.91.116.10:4081 , piece_id=UVTRFM6U5ZOU5BUEGQOR3TCJ6VYZKDQLIAFFGHWQCJNPTUT55H6A, seg_pos={0 0}, upload=19598]: Done uploading piece. err: ecclient: upload failed (node:12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG, address:45.91.116.10:4081): protocol: write tcp 188.166.225.170:54910->45.91.116.10:4081: use of closed network connection; write tcp 188.166.225.170:54910->45.91.116.10:4081: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54910->45.91.116.10:4081: use of closed network connection Aug 9 12:12:10.688309995 (18h46m5.119390137s) [seg_pos={0 0}, upload=19598]: Done waiting for segment. successful: 3 optimal: 13 errs: <nil> Aug 9 12:12:10.917247730 (18h46m5.348327861s) [upload=19598]: Done uploading segments... Aug 9 12:12:10.917278388 (18h46m5.34835854s) [upload=19598]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 3) Duration: 30.231702456s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 3 out of 60 is failed (threshold=3) [...............X..................X.........X...............] duration: 1.855311084s FAIL (2025-08-09 12:11:55 UTC) orbiter warning: operation: ListObjects ======ERROR====== unable to find key in bucket: sj://uptime-checks-placement-20/orbiter_us_0/1K ======LOGS====== Aug 9 11:59:31.819436107 (18h33m26.250516238s) [upload=19388]: Starting upload Aug 9 11:59:31.819492355 (18h33m26.250572516s) [upload=19388]: Done starting upload Aug 9 11:59:31.819563080 (18h33m26.250643211s) [upload=19388]: Uploading segments... Aug 9 11:59:31.819616853 (18h33m26.250696984s) [upload=19388]: Got next segment. Inline: true Aug 9 11:59:31.819621552 (18h33m26.250701683s) [upload=19388]: Waiting for error group managing segments... Aug 9 11:59:31.819660466 (18h33m26.250740597s) [upload=19388]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 9 11:59:31.819665206 (18h33m26.250745327s) [upload=19388]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 9 11:59:31.819667851 (18h33m26.250747972s) [upload=19388]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 9 11:59:32.242597809 (18h33m26.67367794s) [upload=19388]: Done uploading segments... Aug 9 11:59:32.242615142 (18h33m26.673695263s) [upload=19388]: Upload finished. err: <nil> Duration:* 1.8553562s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 4 out of 60 is failed (threshold=3) [.............................X............X.X........X......] duration: 5.903211012s FAIL (2025-08-09 12:11:25 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: too many requests ======LOGS====== Aug 9 12:06:20.421719324 (18h40m14.852799455s) [upload=19509]: Starting upload Aug 9 12:06:20.421777855 (18h40m14.852857987s) [upload=19509]: Done starting upload Aug 9 12:06:20.421895961 (18h40m14.852976092s) [upload=19509]: Uploading segments... Aug 9 12:06:20.422028174 (18h40m14.853108305s) [upload=19509]: Got next segment. Inline: false Aug 9 12:06:20.422073501 (18h40m14.853153622s) [upload=19509]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 9 12:06:20.422077549 (18h40m14.85315768s) [upload=19509]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 9 12:06:20.634384816 (18h40m15.065464937s) [upload=19509]: Next segment err: metaclient: Too Many Requests Aug 9 12:06:20.634410485 (18h40m15.065490606s) [upload=19509]: Done uploading segments... Aug 9 12:06:20.634426876 (18h40m15.065507007s) [upload=19509]: Upload finished. err: metaclient: Too Many Requests Duration: 5.903233237s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 3 out of 60 is failed (threshold=3) [................X..................X.........X..............] duration: 2.003001306s FAIL (2025-08-09 12:11:11 UTC) orbiter warning: operation: ListObjects ======ERROR====== unable to find key in bucket: sj://uptime-checks-placement-20/orbiter_us_0/1K ======LOGS====== Aug 9 11:59:31.819436107 (18h33m26.250516238s) [upload=19388]: Starting upload Aug 9 11:59:31.819492355 (18h33m26.250572516s) [upload=19388]: Done starting upload Aug 9 11:59:31.819563080 (18h33m26.250643211s) [upload=19388]: Uploading segments... Aug 9 11:59:31.819616853 (18h33m26.250696984s) [upload=19388]: Got next segment. Inline: true Aug 9 11:59:31.819621552 (18h33m26.250701683s) [upload=19388]: Waiting for error group managing segments... Aug 9 11:59:31.819660466 (18h33m26.250740597s) [upload=19388]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 9 11:59:31.819665206 (18h33m26.250745327s) [upload=19388]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 9 11:59:31.819667851 (18h33m26.250747972s) [upload=19388]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 9 11:59:32.242597809 (18h33m26.67367794s) [upload=19388]: Done uploading segments... Aug 9 11:59:32.242615142 (18h33m26.673695263s) [upload=19388]: Upload finished. err: <nil> Duration:* 2.003029021s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............X............................................] duration: 5.800102274s FAIL (2025-08-09 12:10:56 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 6) ======LOGS====== ADHZNIB6LBP5VQHFEGSYM3YNNZWNM2N4UU7JTCQ, seg_pos={0 0}, upload=18998]: Uploading piece... Aug 9 11:37:25.005421531 (18h11m19.436501662s) [noise=true , address=45.91.116.4:4940 , piece_id=M33ZGUCEF23HDGQZE5F5KLWQ43NHC7E7LKOVVGQ6P6TM2HFGB4WQ, seg_pos={0 0}, upload=18998]: Uploading piece... Aug 9 11:37:25.005614329 (18h11m19.43669446s) [noise=true , address=45.91.116.8:4047 , piece_id=B27B2QKZXICP4DHTDTCYSZZ7RN2PPCMKSRQBKRK5JJ4BE4RNWQAQ, seg_pos={0 0}, upload=18998]: Uploading piece... Aug 9 11:37:25.005727536 (18h11m19.436807667s) [noise=true , address=45.91.116.12:4225 , piece_id=VFK2SMVTBT76D7YLQWGRA6TF47JD3B7J7F5NFR4E2MQVNGNHYTAQ, seg_pos={0 0}, upload=18998]: Uploading piece... Aug 9 11:37:25.006281517 (18h11m19.437361648s) [noise=true , address=104.36.82.7:4909 , piece_id=OGG5ZCXSLIM5YKMM7K6KLKLWGOTX376DUTDT3567OB3AJIBPBKDQ, seg_pos={0 0}, upload=18998]: Uploading piece... Aug 9 11:37:25.006632558 (18h11m19.437712679s) [noise=true , address=45.91.116.13:4155 , piece_id=K4CGD4OCR2B6ZWAZHB5MDBECVJB3AC2HDXS2TUVAVGIWQYJEARLA, seg_pos={0 0}, upload=18998]: Uploading piece... Aug 9 11:37:25.006996264 (18h11m19.438076395s) [noise=true , address=77.90.31.18:4806 , piece_id=QCURQYEJL22LBT2YFBR62RIHH7VNSGL7GERP66GKHXK3JP5WLVZQ, seg_pos={0 0}, upload=18998]: Uploading piece... Aug 9 11:37:25.007245231 (18h11m19.438325362s) [noise=true , address=104.36.82.3:4765 , piece_id=HBLTTPZYK5ZXNG2XV53QVJA3NPX3YWKEAAAFVMDXUO6BLR3QKCWQ, seg_pos={0 0}, upload=18998]: Uploading piece... Aug 9 11:37:25.007475481 (18h11m19.438555612s) [noise=true , address=45.91.116.6:4973 , piece_id=MHDNXTVKFMLRKGS7JIPL4U5VQMBXGP4EYM7JEGPSK4U3GIJ5JEHQ, seg_pos={0 0}, upload=18998]: Uploading piece... Aug 9 11:37:25.007873984 (18h11m19.438954115s) [noise=true , address=45.91.116.7:4010 , piece_id=5G3O552GD5XO7SNIHST623QJBWJF2C7B6EN7ISGHHY3FYV6QL2DQ, seg_pos={0 0}, upload=18998]: Uploading piece... Aug 9 11:37:26.733124951 (18h11m21.164205082s) [upload=18998]: Next returned nil segment Aug 9 11:37:26.733142194 (18h11m21.164222325s) [upload=18998]: Waiting for error group managing segments... Aug 9 11:37:27.100757697 (18h11m21.531837848s) [noise=true , address=104.36.82.14:4222 , piece_id=Z4X6RJ5HBAYGUNKT6DXWSS2TEG2OUFPMH3NWYGA7D26XIFH46UAA, seg_pos={0 0}, upload=18998]: Done uploading piece. err: <nil> Aug 9 11:37:27.106781913 (18h11m21.537862064s) [noise=true , address=104.36.82.7:4912 , piece_id=MRA5X6N6MDQ2TADHZNIB6LBP5VQHFEGSYM3YNNZWNM2N4UU7JTCQ, seg_pos={0 0}, upload=18998]: Done uploading piece. err: <nil> Aug 9 11:37:27.136679105 (18h11m21.567759236s) [noise=true , address=104.36.82.5:4838 , piece_id=T3WCWCLB4WAXQY3S6VCFHLKYK6IY2Q6P2776CLGYS4QOFTXMYVEQ, seg_pos={0 0}, upload=18998]: Done uploading piece. err: <nil> Aug 9 11:37:27.139126447 (18h11m21.570206578s) [noise=true , address=104.36.82.3:4765 , piece_id=HBLTTPZYK5ZXNG2XV53QVJA3NPX3YWKEAAAFVMDXUO6BLR3QKCWQ, seg_pos={0 0}, upload=18998]: Done uploading piece. err: <nil> Aug 9 11:37:27.183514474 (18h11m21.614594606s) [noise=true , address=104.36.82.7:4909 , piece_id=OGG5ZCXSLIM5YKMM7K6KLKLWGOTX376DUTDT3567OB3AJIBPBKDQ, seg_pos={0 0}, upload=18998]: Done uploading piece. err: <nil> Aug 9 11:37:53.489259032 (18h11m47.920339163s) [noise=true , address=45.91.116.13:4155 , piece_id=K4CGD4OCR2B6ZWAZHB5MDBECVJB3AC2HDXS2TUVAVGIWQYJEARLA, seg_pos={0 0}, upload=18998]: Done uploading piece. err: <nil> Aug 9 11:37:54.124525138 (18h11m48.555605259s) [noise=true , address=45.91.116.7:4010 , piece_id=5G3O552GD5XO7SNIHST623QJBWJF2C7B6EN7ISGHHY3FYV6QL2DQ, seg_pos={0 0}, upload=18998]: Done uploading piece. err: ecclient: upload failed (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w, address:45.91.116.7:4010): protocol: write tcp 188.166.225.170:54960->45.91.116.7:4010: use of closed network connection; write tcp 188.166.225.170:54960->45.91.116.7:4010: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54960->45.91.116.7:4010: use of closed network connection Aug 9 11:37:54.124581035 (18h11m48.555661166s) [noise=true , address=77.90.31.10:4520 , piece_id=YRBBWM54FGU3B74JQ7EOAO7VSGVFXAVT4SNZ6KN3IQFIHIEVV5HA, seg_pos={0 0}, upload=18998]: Done uploading piece. err: ecclient: upload failed (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp, address:77.90.31.10:4520): protocol: write tcp 188.166.225.170:39560->77.90.31.10:4520: use of closed network connection; write tcp 188.166.225.170:39560->77.90.31.10:4520: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39560->77.90.31.10:4520: use of closed network connection Aug 9 11:37:54.124615480 (18h11m48.555695652s) [noise=true , address=45.91.116.12:4225 , piece_id=VFK2SMVTBT76D7YLQWGRA6TF47JD3B7J7F5NFR4E2MQVNGNHYTAQ, seg_pos={0 0}, upload=18998]: Done uploading piece. err: ecclient: upload failed (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy, address:45.91.116.12:4225): protocol: write tcp 188.166.225.170:38054->45.91.116.12:4225: use of closed network connection; write tcp 188.166.225.170:38054->45.91.116.12:4225: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38054->45.91.116.12:4225: use of closed network connection Aug 9 11:37:54.124643875 (18h11m48.555724016s) [noise=true , address=45.91.116.4:4940 , piece_id=M33ZGUCEF23HDGQZE5F5KLWQ43NHC7E7LKOVVGQ6P6TM2HFGB4WQ, seg_pos={0 0}, upload=18998]: Done uploading piece. err: ecclient: upload failed (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe, address:45.91.116.4:4940): protocol: write tcp 188.166.225.170:40630->45.91.116.4:4940: use of closed network connection; write tcp 188.166.225.170:40630->45.91.116.4:4940: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40630->45.91.116.4:4940: use of closed network connection Aug 9 11:37:54.124693159 (18h11m48.55577335s) [noise=true , address=45.91.116.14:4189 , piece_id=XVQMTYHB6H3IKSERYT2BEB4NP42KV6FRE6J2MKKU5WEOCVBHYIWA, seg_pos={0 0}, upload=18998]: Done uploading piece. err: ecclient: upload failed (node:1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N, address:45.91.116.14:4189): protocol: write tcp 188.166.225.170:37650->45.91.116.14:4189: use of closed network connection; write tcp 188.166.225.170:37650->45.91.116.14:4189: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37650->45.91.116.14:4189: use of closed network connection Aug 9 11:37:54.124771740 (18h11m48.555851871s) [noise=true , address=77.90.31.18:4806 , piece_id=QCURQYEJL22LBT2YFBR62RIHH7VNSGL7GERP66GKHXK3JP5WLVZQ, seg_pos={0 0}, upload=18998]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: write tcp 188.166.225.170:55536->77.90.31.18:4806: use of closed network connection; write tcp 188.166.225.170:55536->77.90.31.18:4806: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55536->77.90.31.18:4806: use of closed network connection Aug 9 11:37:54.124817217 (18h11m48.555897348s) [noise=true , address=77.90.31.15:4697 , piece_id=Q26SRGZO2YBJWPDIG4U422EICE6XEXPLHFHXLKFUBKMMJAZ4W7UQ, seg_pos={0 0}, upload=18998]: Done uploading piece. err: ecclient: upload failed (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, address:77.90.31.15:4697): protocol: write tcp 188.166.225.170:49618->77.90.31.15:4697: use of closed network connection; write tcp 188.166.225.170:49618->77.90.31.15:4697: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49618->77.90.31.15:4697: use of closed network connection Aug 9 11:37:54.124833387 (18h11m48.555913559s) [noise=true , address=45.91.116.6:4973 , piece_id=MHDNXTVKFMLRKGS7JIPL4U5VQMBXGP4EYM7JEGPSK4U3GIJ5JEHQ, seg_pos={0 0}, upload=18998]: Done uploading piece. err: ecclient: upload failed (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, address:45.91.116.6:4973): protocol: write tcp 188.166.225.170:59438->45.91.116.6:4973: use of closed network connection; write tcp 188.166.225.170:59438->45.91.116.6:4973: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59438->45.91.116.6:4973: use of closed network connection Aug 9 11:37:54.124840752 (18h11m48.555920883s) [noise=true , address=77.90.31.17:4772 , piece_id=MGZPHOODRIOTGABQJHNIP5FGF4XN6A723ZHUBKI6KNAJDKFWWOJA, seg_pos={0 0}, upload=18998]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: write tcp 188.166.225.170:42704->77.90.31.17:4772: use of closed network connection; write tcp 188.166.225.170:42704->77.90.31.17:4772: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42704->77.90.31.17:4772: use of closed network connection Aug 9 11:37:54.124845251 (18h11m48.555925382s) [noise=true , address=45.91.116.8:4047 , piece_id=B27B2QKZXICP4DHTDTCYSZZ7RN2PPCMKSRQBKRK5JJ4BE4RNWQAQ, seg_pos={0 0}, upload=18998]: Done uploading piece. err: ecclient: upload failed (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN, address:45.91.116.8:4047): protocol: write tcp 188.166.225.170:40794->45.91.116.8:4047: use of closed network connection; write tcp 188.166.225.170:40794->45.91.116.8:4047: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40794->45.91.116.8:4047: use of closed network connection Aug 9 11:37:54.124866321 (18h11m48.555946492s) [seg_pos={0 0}, upload=18998]: Done waiting for segment. successful: 6 optimal: 13 errs: <nil> Aug 9 11:37:54.353990811 (18h11m48.785070942s) [upload=18998]: Done uploading segments... Aug 9 11:37:54.354028673 (18h11m48.785108804s) [upload=18998]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 6) Duration: 5.800127655s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 3 out of 60 is failed (threshold=3) [.................X..................X.........X.............] duration: 1.917792058s FAIL (2025-08-09 12:10:29 UTC) orbiter warning: operation: ListObjects ======ERROR====== unable to find key in bucket: sj://uptime-checks-placement-20/orbiter_us_0/1K ======LOGS====== Aug 9 11:59:31.819436107 (18h33m26.250516238s) [upload=19388]: Starting upload Aug 9 11:59:31.819492355 (18h33m26.250572516s) [upload=19388]: Done starting upload Aug 9 11:59:31.819563080 (18h33m26.250643211s) [upload=19388]: Uploading segments... Aug 9 11:59:31.819616853 (18h33m26.250696984s) [upload=19388]: Got next segment. Inline: true Aug 9 11:59:31.819621552 (18h33m26.250701683s) [upload=19388]: Waiting for error group managing segments... Aug 9 11:59:31.819660466 (18h33m26.250740597s) [upload=19388]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 9 11:59:31.819665206 (18h33m26.250745327s) [upload=19388]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 9 11:59:31.819667851 (18h33m26.250747972s) [upload=19388]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 9 11:59:32.242597809 (18h33m26.67367794s) [upload=19388]: Done uploading segments... Aug 9 11:59:32.242615142 (18h33m26.673695263s) [upload=19388]: Upload finished. err: <nil> Duration:* 1.917811055s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 4 out of 60 is failed (threshold=3) [..............................X............X.X........X.....] duration: 5.975293508s FAIL (2025-08-09 12:10:27 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: too many requests ======LOGS====== Aug 9 12:06:20.421719324 (18h40m14.852799455s) [upload=19509]: Starting upload Aug 9 12:06:20.421777855 (18h40m14.852857987s) [upload=19509]: Done starting upload Aug 9 12:06:20.421895961 (18h40m14.852976092s) [upload=19509]: Uploading segments... Aug 9 12:06:20.422028174 (18h40m14.853108305s) [upload=19509]: Got next segment. Inline: false Aug 9 12:06:20.422073501 (18h40m14.853153622s) [upload=19509]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 9 12:06:20.422077549 (18h40m14.85315768s) [upload=19509]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 9 12:06:20.634384816 (18h40m15.065464937s) [upload=19509]: Next segment err: metaclient: Too Many Requests Aug 9 12:06:20.634410485 (18h40m15.065490606s) [upload=19509]: Done uploading segments... Aug 9 12:06:20.634426876 (18h40m15.065507007s) [upload=19509]: Upload finished. err: metaclient: Too Many Requests Duration: 5.97531498s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................X...........................................] duration: 5.788239809s FAIL (2025-08-09 12:10:15 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 6) ======LOGS====== ADHZNIB6LBP5VQHFEGSYM3YNNZWNM2N4UU7JTCQ, seg_pos={0 0}, upload=18998]: Uploading piece... Aug 9 11:37:25.005421531 (18h11m19.436501662s) [noise=true , address=45.91.116.4:4940 , piece_id=M33ZGUCEF23HDGQZE5F5KLWQ43NHC7E7LKOVVGQ6P6TM2HFGB4WQ, seg_pos={0 0}, upload=18998]: Uploading piece... Aug 9 11:37:25.005614329 (18h11m19.43669446s) [noise=true , address=45.91.116.8:4047 , piece_id=B27B2QKZXICP4DHTDTCYSZZ7RN2PPCMKSRQBKRK5JJ4BE4RNWQAQ, seg_pos={0 0}, upload=18998]: Uploading piece... Aug 9 11:37:25.005727536 (18h11m19.436807667s) [noise=true , address=45.91.116.12:4225 , piece_id=VFK2SMVTBT76D7YLQWGRA6TF47JD3B7J7F5NFR4E2MQVNGNHYTAQ, seg_pos={0 0}, upload=18998]: Uploading piece... Aug 9 11:37:25.006281517 (18h11m19.437361648s) [noise=true , address=104.36.82.7:4909 , piece_id=OGG5ZCXSLIM5YKMM7K6KLKLWGOTX376DUTDT3567OB3AJIBPBKDQ, seg_pos={0 0}, upload=18998]: Uploading piece... Aug 9 11:37:25.006632558 (18h11m19.437712679s) [noise=true , address=45.91.116.13:4155 , piece_id=K4CGD4OCR2B6ZWAZHB5MDBECVJB3AC2HDXS2TUVAVGIWQYJEARLA, seg_pos={0 0}, upload=18998]: Uploading piece... Aug 9 11:37:25.006996264 (18h11m19.438076395s) [noise=true , address=77.90.31.18:4806 , piece_id=QCURQYEJL22LBT2YFBR62RIHH7VNSGL7GERP66GKHXK3JP5WLVZQ, seg_pos={0 0}, upload=18998]: Uploading piece... Aug 9 11:37:25.007245231 (18h11m19.438325362s) [noise=true , address=104.36.82.3:4765 , piece_id=HBLTTPZYK5ZXNG2XV53QVJA3NPX3YWKEAAAFVMDXUO6BLR3QKCWQ, seg_pos={0 0}, upload=18998]: Uploading piece... Aug 9 11:37:25.007475481 (18h11m19.438555612s) [noise=true , address=45.91.116.6:4973 , piece_id=MHDNXTVKFMLRKGS7JIPL4U5VQMBXGP4EYM7JEGPSK4U3GIJ5JEHQ, seg_pos={0 0}, upload=18998]: Uploading piece... Aug 9 11:37:25.007873984 (18h11m19.438954115s) [noise=true , address=45.91.116.7:4010 , piece_id=5G3O552GD5XO7SNIHST623QJBWJF2C7B6EN7ISGHHY3FYV6QL2DQ, seg_pos={0 0}, upload=18998]: Uploading piece... Aug 9 11:37:26.733124951 (18h11m21.164205082s) [upload=18998]: Next returned nil segment Aug 9 11:37:26.733142194 (18h11m21.164222325s) [upload=18998]: Waiting for error group managing segments... Aug 9 11:37:27.100757697 (18h11m21.531837848s) [noise=true , address=104.36.82.14:4222 , piece_id=Z4X6RJ5HBAYGUNKT6DXWSS2TEG2OUFPMH3NWYGA7D26XIFH46UAA, seg_pos={0 0}, upload=18998]: Done uploading piece. err: <nil> Aug 9 11:37:27.106781913 (18h11m21.537862064s) [noise=true , address=104.36.82.7:4912 , piece_id=MRA5X6N6MDQ2TADHZNIB6LBP5VQHFEGSYM3YNNZWNM2N4UU7JTCQ, seg_pos={0 0}, upload=18998]: Done uploading piece. err: <nil> Aug 9 11:37:27.136679105 (18h11m21.567759236s) [noise=true , address=104.36.82.5:4838 , piece_id=T3WCWCLB4WAXQY3S6VCFHLKYK6IY2Q6P2776CLGYS4QOFTXMYVEQ, seg_pos={0 0}, upload=18998]: Done uploading piece. err: <nil> Aug 9 11:37:27.139126447 (18h11m21.570206578s) [noise=true , address=104.36.82.3:4765 , piece_id=HBLTTPZYK5ZXNG2XV53QVJA3NPX3YWKEAAAFVMDXUO6BLR3QKCWQ, seg_pos={0 0}, upload=18998]: Done uploading piece. err: <nil> Aug 9 11:37:27.183514474 (18h11m21.614594606s) [noise=true , address=104.36.82.7:4909 , piece_id=OGG5ZCXSLIM5YKMM7K6KLKLWGOTX376DUTDT3567OB3AJIBPBKDQ, seg_pos={0 0}, upload=18998]: Done uploading piece. err: <nil> Aug 9 11:37:53.489259032 (18h11m47.920339163s) [noise=true , address=45.91.116.13:4155 , piece_id=K4CGD4OCR2B6ZWAZHB5MDBECVJB3AC2HDXS2TUVAVGIWQYJEARLA, seg_pos={0 0}, upload=18998]: Done uploading piece. err: <nil> Aug 9 11:37:54.124525138 (18h11m48.555605259s) [noise=true , address=45.91.116.7:4010 , piece_id=5G3O552GD5XO7SNIHST623QJBWJF2C7B6EN7ISGHHY3FYV6QL2DQ, seg_pos={0 0}, upload=18998]: Done uploading piece. err: ecclient: upload failed (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w, address:45.91.116.7:4010): protocol: write tcp 188.166.225.170:54960->45.91.116.7:4010: use of closed network connection; write tcp 188.166.225.170:54960->45.91.116.7:4010: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54960->45.91.116.7:4010: use of closed network connection Aug 9 11:37:54.124581035 (18h11m48.555661166s) [noise=true , address=77.90.31.10:4520 , piece_id=YRBBWM54FGU3B74JQ7EOAO7VSGVFXAVT4SNZ6KN3IQFIHIEVV5HA, seg_pos={0 0}, upload=18998]: Done uploading piece. err: ecclient: upload failed (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp, address:77.90.31.10:4520): protocol: write tcp 188.166.225.170:39560->77.90.31.10:4520: use of closed network connection; write tcp 188.166.225.170:39560->77.90.31.10:4520: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39560->77.90.31.10:4520: use of closed network connection Aug 9 11:37:54.124615480 (18h11m48.555695652s) [noise=true , address=45.91.116.12:4225 , piece_id=VFK2SMVTBT76D7YLQWGRA6TF47JD3B7J7F5NFR4E2MQVNGNHYTAQ, seg_pos={0 0}, upload=18998]: Done uploading piece. err: ecclient: upload failed (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy, address:45.91.116.12:4225): protocol: write tcp 188.166.225.170:38054->45.91.116.12:4225: use of closed network connection; write tcp 188.166.225.170:38054->45.91.116.12:4225: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38054->45.91.116.12:4225: use of closed network connection Aug 9 11:37:54.124643875 (18h11m48.555724016s) [noise=true , address=45.91.116.4:4940 , piece_id=M33ZGUCEF23HDGQZE5F5KLWQ43NHC7E7LKOVVGQ6P6TM2HFGB4WQ, seg_pos={0 0}, upload=18998]: Done uploading piece. err: ecclient: upload failed (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe, address:45.91.116.4:4940): protocol: write tcp 188.166.225.170:40630->45.91.116.4:4940: use of closed network connection; write tcp 188.166.225.170:40630->45.91.116.4:4940: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40630->45.91.116.4:4940: use of closed network connection Aug 9 11:37:54.124693159 (18h11m48.55577335s) [noise=true , address=45.91.116.14:4189 , piece_id=XVQMTYHB6H3IKSERYT2BEB4NP42KV6FRE6J2MKKU5WEOCVBHYIWA, seg_pos={0 0}, upload=18998]: Done uploading piece. err: ecclient: upload failed (node:1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N, address:45.91.116.14:4189): protocol: write tcp 188.166.225.170:37650->45.91.116.14:4189: use of closed network connection; write tcp 188.166.225.170:37650->45.91.116.14:4189: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37650->45.91.116.14:4189: use of closed network connection Aug 9 11:37:54.124771740 (18h11m48.555851871s) [noise=true , address=77.90.31.18:4806 , piece_id=QCURQYEJL22LBT2YFBR62RIHH7VNSGL7GERP66GKHXK3JP5WLVZQ, seg_pos={0 0}, upload=18998]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: write tcp 188.166.225.170:55536->77.90.31.18:4806: use of closed network connection; write tcp 188.166.225.170:55536->77.90.31.18:4806: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55536->77.90.31.18:4806: use of closed network connection Aug 9 11:37:54.124817217 (18h11m48.555897348s) [noise=true , address=77.90.31.15:4697 , piece_id=Q26SRGZO2YBJWPDIG4U422EICE6XEXPLHFHXLKFUBKMMJAZ4W7UQ, seg_pos={0 0}, upload=18998]: Done uploading piece. err: ecclient: upload failed (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, address:77.90.31.15:4697): protocol: write tcp 188.166.225.170:49618->77.90.31.15:4697: use of closed network connection; write tcp 188.166.225.170:49618->77.90.31.15:4697: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49618->77.90.31.15:4697: use of closed network connection Aug 9 11:37:54.124833387 (18h11m48.555913559s) [noise=true , address=45.91.116.6:4973 , piece_id=MHDNXTVKFMLRKGS7JIPL4U5VQMBXGP4EYM7JEGPSK4U3GIJ5JEHQ, seg_pos={0 0}, upload=18998]: Done uploading piece. err: ecclient: upload failed (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, address:45.91.116.6:4973): protocol: write tcp 188.166.225.170:59438->45.91.116.6:4973: use of closed network connection; write tcp 188.166.225.170:59438->45.91.116.6:4973: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59438->45.91.116.6:4973: use of closed network connection Aug 9 11:37:54.124840752 (18h11m48.555920883s) [noise=true , address=77.90.31.17:4772 , piece_id=MGZPHOODRIOTGABQJHNIP5FGF4XN6A723ZHUBKI6KNAJDKFWWOJA, seg_pos={0 0}, upload=18998]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: write tcp 188.166.225.170:42704->77.90.31.17:4772: use of closed network connection; write tcp 188.166.225.170:42704->77.90.31.17:4772: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42704->77.90.31.17:4772: use of closed network connection Aug 9 11:37:54.124845251 (18h11m48.555925382s) [noise=true , address=45.91.116.8:4047 , piece_id=B27B2QKZXICP4DHTDTCYSZZ7RN2PPCMKSRQBKRK5JJ4BE4RNWQAQ, seg_pos={0 0}, upload=18998]: Done uploading piece. err: ecclient: upload failed (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN, address:45.91.116.8:4047): protocol: write tcp 188.166.225.170:40794->45.91.116.8:4047: use of closed network connection; write tcp 188.166.225.170:40794->45.91.116.8:4047: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40794->45.91.116.8:4047: use of closed network connection Aug 9 11:37:54.124866321 (18h11m48.555946492s) [seg_pos={0 0}, upload=18998]: Done waiting for segment. successful: 6 optimal: 13 errs: <nil> Aug 9 11:37:54.353990811 (18h11m48.785070942s) [upload=18998]: Done uploading segments... Aug 9 11:37:54.354028673 (18h11m48.785108804s) [upload=18998]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 6) Duration: 5.788268144s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 4 out of 60 is failed (threshold=3) [...............................X............X.X........X....] duration: 5.812629935s FAIL (2025-08-09 12:09:51 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: too many requests ======LOGS====== Aug 9 12:06:20.421719324 (18h40m14.852799455s) [upload=19509]: Starting upload Aug 9 12:06:20.421777855 (18h40m14.852857987s) [upload=19509]: Done starting upload Aug 9 12:06:20.421895961 (18h40m14.852976092s) [upload=19509]: Uploading segments... Aug 9 12:06:20.422028174 (18h40m14.853108305s) [upload=19509]: Got next segment. Inline: false Aug 9 12:06:20.422073501 (18h40m14.853153622s) [upload=19509]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 9 12:06:20.422077549 (18h40m14.85315768s) [upload=19509]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 9 12:06:20.634384816 (18h40m15.065464937s) [upload=19509]: Next segment err: metaclient: Too Many Requests Aug 9 12:06:20.634410485 (18h40m15.065490606s) [upload=19509]: Done uploading segments... Aug 9 12:06:20.634426876 (18h40m15.065507007s) [upload=19509]: Upload finished. err: metaclient: Too Many Requests Duration: 5.812657068s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 3 out of 60 is failed (threshold=3) [..................X..................X.........X............] duration: 2.007008752s FAIL (2025-08-09 12:09:26 UTC) orbiter warning: operation: ListObjects ======ERROR====== unable to find key in bucket: sj://uptime-checks-placement-20/orbiter_us_0/1K ======LOGS====== Aug 9 11:59:31.819436107 (18h33m26.250516238s) [upload=19388]: Starting upload Aug 9 11:59:31.819492355 (18h33m26.250572516s) [upload=19388]: Done starting upload Aug 9 11:59:31.819563080 (18h33m26.250643211s) [upload=19388]: Uploading segments... Aug 9 11:59:31.819616853 (18h33m26.250696984s) [upload=19388]: Got next segment. Inline: true Aug 9 11:59:31.819621552 (18h33m26.250701683s) [upload=19388]: Waiting for error group managing segments... Aug 9 11:59:31.819660466 (18h33m26.250740597s) [upload=19388]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 9 11:59:31.819665206 (18h33m26.250745327s) [upload=19388]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 9 11:59:31.819667851 (18h33m26.250747972s) [upload=19388]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 9 11:59:32.242597809 (18h33m26.67367794s) [upload=19388]: Done uploading segments... Aug 9 11:59:32.242615142 (18h33m26.673695263s) [upload=19388]: Upload finished. err: <nil> Duration:* 2.007057394s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................X..........................................] duration: 5.866152148s FAIL (2025-08-09 12:09:12 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 6) ======LOGS====== ADHZNIB6LBP5VQHFEGSYM3YNNZWNM2N4UU7JTCQ, seg_pos={0 0}, upload=18998]: Uploading piece... Aug 9 11:37:25.005421531 (18h11m19.436501662s) [noise=true , address=45.91.116.4:4940 , piece_id=M33ZGUCEF23HDGQZE5F5KLWQ43NHC7E7LKOVVGQ6P6TM2HFGB4WQ, seg_pos={0 0}, upload=18998]: Uploading piece... Aug 9 11:37:25.005614329 (18h11m19.43669446s) [noise=true , address=45.91.116.8:4047 , piece_id=B27B2QKZXICP4DHTDTCYSZZ7RN2PPCMKSRQBKRK5JJ4BE4RNWQAQ, seg_pos={0 0}, upload=18998]: Uploading piece... Aug 9 11:37:25.005727536 (18h11m19.436807667s) [noise=true , address=45.91.116.12:4225 , piece_id=VFK2SMVTBT76D7YLQWGRA6TF47JD3B7J7F5NFR4E2MQVNGNHYTAQ, seg_pos={0 0}, upload=18998]: Uploading piece... Aug 9 11:37:25.006281517 (18h11m19.437361648s) [noise=true , address=104.36.82.7:4909 , piece_id=OGG5ZCXSLIM5YKMM7K6KLKLWGOTX376DUTDT3567OB3AJIBPBKDQ, seg_pos={0 0}, upload=18998]: Uploading piece... Aug 9 11:37:25.006632558 (18h11m19.437712679s) [noise=true , address=45.91.116.13:4155 , piece_id=K4CGD4OCR2B6ZWAZHB5MDBECVJB3AC2HDXS2TUVAVGIWQYJEARLA, seg_pos={0 0}, upload=18998]: Uploading piece... Aug 9 11:37:25.006996264 (18h11m19.438076395s) [noise=true , address=77.90.31.18:4806 , piece_id=QCURQYEJL22LBT2YFBR62RIHH7VNSGL7GERP66GKHXK3JP5WLVZQ, seg_pos={0 0}, upload=18998]: Uploading piece... Aug 9 11:37:25.007245231 (18h11m19.438325362s) [noise=true , address=104.36.82.3:4765 , piece_id=HBLTTPZYK5ZXNG2XV53QVJA3NPX3YWKEAAAFVMDXUO6BLR3QKCWQ, seg_pos={0 0}, upload=18998]: Uploading piece... Aug 9 11:37:25.007475481 (18h11m19.438555612s) [noise=true , address=45.91.116.6:4973 , piece_id=MHDNXTVKFMLRKGS7JIPL4U5VQMBXGP4EYM7JEGPSK4U3GIJ5JEHQ, seg_pos={0 0}, upload=18998]: Uploading piece... Aug 9 11:37:25.007873984 (18h11m19.438954115s) [noise=true , address=45.91.116.7:4010 , piece_id=5G3O552GD5XO7SNIHST623QJBWJF2C7B6EN7ISGHHY3FYV6QL2DQ, seg_pos={0 0}, upload=18998]: Uploading piece... Aug 9 11:37:26.733124951 (18h11m21.164205082s) [upload=18998]: Next returned nil segment Aug 9 11:37:26.733142194 (18h11m21.164222325s) [upload=18998]: Waiting for error group managing segments... Aug 9 11:37:27.100757697 (18h11m21.531837848s) [noise=true , address=104.36.82.14:4222 , piece_id=Z4X6RJ5HBAYGUNKT6DXWSS2TEG2OUFPMH3NWYGA7D26XIFH46UAA, seg_pos={0 0}, upload=18998]: Done uploading piece. err: <nil> Aug 9 11:37:27.106781913 (18h11m21.537862064s) [noise=true , address=104.36.82.7:4912 , piece_id=MRA5X6N6MDQ2TADHZNIB6LBP5VQHFEGSYM3YNNZWNM2N4UU7JTCQ, seg_pos={0 0}, upload=18998]: Done uploading piece. err: <nil> Aug 9 11:37:27.136679105 (18h11m21.567759236s) [noise=true , address=104.36.82.5:4838 , piece_id=T3WCWCLB4WAXQY3S6VCFHLKYK6IY2Q6P2776CLGYS4QOFTXMYVEQ, seg_pos={0 0}, upload=18998]: Done uploading piece. err: <nil> Aug 9 11:37:27.139126447 (18h11m21.570206578s) [noise=true , address=104.36.82.3:4765 , piece_id=HBLTTPZYK5ZXNG2XV53QVJA3NPX3YWKEAAAFVMDXUO6BLR3QKCWQ, seg_pos={0 0}, upload=18998]: Done uploading piece. err: <nil> Aug 9 11:37:27.183514474 (18h11m21.614594606s) [noise=true , address=104.36.82.7:4909 , piece_id=OGG5ZCXSLIM5YKMM7K6KLKLWGOTX376DUTDT3567OB3AJIBPBKDQ, seg_pos={0 0}, upload=18998]: Done uploading piece. err: <nil> Aug 9 11:37:53.489259032 (18h11m47.920339163s) [noise=true , address=45.91.116.13:4155 , piece_id=K4CGD4OCR2B6ZWAZHB5MDBECVJB3AC2HDXS2TUVAVGIWQYJEARLA, seg_pos={0 0}, upload=18998]: Done uploading piece. err: <nil> Aug 9 11:37:54.124525138 (18h11m48.555605259s) [noise=true , address=45.91.116.7:4010 , piece_id=5G3O552GD5XO7SNIHST623QJBWJF2C7B6EN7ISGHHY3FYV6QL2DQ, seg_pos={0 0}, upload=18998]: Done uploading piece. err: ecclient: upload failed (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w, address:45.91.116.7:4010): protocol: write tcp 188.166.225.170:54960->45.91.116.7:4010: use of closed network connection; write tcp 188.166.225.170:54960->45.91.116.7:4010: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54960->45.91.116.7:4010: use of closed network connection Aug 9 11:37:54.124581035 (18h11m48.555661166s) [noise=true , address=77.90.31.10:4520 , piece_id=YRBBWM54FGU3B74JQ7EOAO7VSGVFXAVT4SNZ6KN3IQFIHIEVV5HA, seg_pos={0 0}, upload=18998]: Done uploading piece. err: ecclient: upload failed (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp, address:77.90.31.10:4520): protocol: write tcp 188.166.225.170:39560->77.90.31.10:4520: use of closed network connection; write tcp 188.166.225.170:39560->77.90.31.10:4520: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39560->77.90.31.10:4520: use of closed network connection Aug 9 11:37:54.124615480 (18h11m48.555695652s) [noise=true , address=45.91.116.12:4225 , piece_id=VFK2SMVTBT76D7YLQWGRA6TF47JD3B7J7F5NFR4E2MQVNGNHYTAQ, seg_pos={0 0}, upload=18998]: Done uploading piece. err: ecclient: upload failed (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy, address:45.91.116.12:4225): protocol: write tcp 188.166.225.170:38054->45.91.116.12:4225: use of closed network connection; write tcp 188.166.225.170:38054->45.91.116.12:4225: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38054->45.91.116.12:4225: use of closed network connection Aug 9 11:37:54.124643875 (18h11m48.555724016s) [noise=true , address=45.91.116.4:4940 , piece_id=M33ZGUCEF23HDGQZE5F5KLWQ43NHC7E7LKOVVGQ6P6TM2HFGB4WQ, seg_pos={0 0}, upload=18998]: Done uploading piece. err: ecclient: upload failed (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe, address:45.91.116.4:4940): protocol: write tcp 188.166.225.170:40630->45.91.116.4:4940: use of closed network connection; write tcp 188.166.225.170:40630->45.91.116.4:4940: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40630->45.91.116.4:4940: use of closed network connection Aug 9 11:37:54.124693159 (18h11m48.55577335s) [noise=true , address=45.91.116.14:4189 , piece_id=XVQMTYHB6H3IKSERYT2BEB4NP42KV6FRE6J2MKKU5WEOCVBHYIWA, seg_pos={0 0}, upload=18998]: Done uploading piece. err: ecclient: upload failed (node:1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N, address:45.91.116.14:4189): protocol: write tcp 188.166.225.170:37650->45.91.116.14:4189: use of closed network connection; write tcp 188.166.225.170:37650->45.91.116.14:4189: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37650->45.91.116.14:4189: use of closed network connection Aug 9 11:37:54.124771740 (18h11m48.555851871s) [noise=true , address=77.90.31.18:4806 , piece_id=QCURQYEJL22LBT2YFBR62RIHH7VNSGL7GERP66GKHXK3JP5WLVZQ, seg_pos={0 0}, upload=18998]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: write tcp 188.166.225.170:55536->77.90.31.18:4806: use of closed network connection; write tcp 188.166.225.170:55536->77.90.31.18:4806: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55536->77.90.31.18:4806: use of closed network connection Aug 9 11:37:54.124817217 (18h11m48.555897348s) [noise=true , address=77.90.31.15:4697 , piece_id=Q26SRGZO2YBJWPDIG4U422EICE6XEXPLHFHXLKFUBKMMJAZ4W7UQ, seg_pos={0 0}, upload=18998]: Done uploading piece. err: ecclient: upload failed (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, address:77.90.31.15:4697): protocol: write tcp 188.166.225.170:49618->77.90.31.15:4697: use of closed network connection; write tcp 188.166.225.170:49618->77.90.31.15:4697: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49618->77.90.31.15:4697: use of closed network connection Aug 9 11:37:54.124833387 (18h11m48.555913559s) [noise=true , address=45.91.116.6:4973 , piece_id=MHDNXTVKFMLRKGS7JIPL4U5VQMBXGP4EYM7JEGPSK4U3GIJ5JEHQ, seg_pos={0 0}, upload=18998]: Done uploading piece. err: ecclient: upload failed (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, address:45.91.116.6:4973): protocol: write tcp 188.166.225.170:59438->45.91.116.6:4973: use of closed network connection; write tcp 188.166.225.170:59438->45.91.116.6:4973: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59438->45.91.116.6:4973: use of closed network connection Aug 9 11:37:54.124840752 (18h11m48.555920883s) [noise=true , address=77.90.31.17:4772 , piece_id=MGZPHOODRIOTGABQJHNIP5FGF4XN6A723ZHUBKI6KNAJDKFWWOJA, seg_pos={0 0}, upload=18998]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: write tcp 188.166.225.170:42704->77.90.31.17:4772: use of closed network connection; write tcp 188.166.225.170:42704->77.90.31.17:4772: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42704->77.90.31.17:4772: use of closed network connection Aug 9 11:37:54.124845251 (18h11m48.555925382s) [noise=true , address=45.91.116.8:4047 , piece_id=B27B2QKZXICP4DHTDTCYSZZ7RN2PPCMKSRQBKRK5JJ4BE4RNWQAQ, seg_pos={0 0}, upload=18998]: Done uploading piece. err: ecclient: upload failed (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN, address:45.91.116.8:4047): protocol: write tcp 188.166.225.170:40794->45.91.116.8:4047: use of closed network connection; write tcp 188.166.225.170:40794->45.91.116.8:4047: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40794->45.91.116.8:4047: use of closed network connection Aug 9 11:37:54.124866321 (18h11m48.555946492s) [seg_pos={0 0}, upload=18998]: Done waiting for segment. successful: 6 optimal: 13 errs: <nil> Aug 9 11:37:54.353990811 (18h11m48.785070942s) [upload=18998]: Done uploading segments... Aug 9 11:37:54.354028673 (18h11m48.785108804s) [upload=18998]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 6) Duration: 5.866183499s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 4 out of 60 is failed (threshold=3) [................................X............X.X........X...] duration: 5.71504695s FAIL (2025-08-09 12:09:06 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: too many requests ======LOGS====== Aug 9 12:06:20.421719324 (18h40m14.852799455s) [upload=19509]: Starting upload Aug 9 12:06:20.421777855 (18h40m14.852857987s) [upload=19509]: Done starting upload Aug 9 12:06:20.421895961 (18h40m14.852976092s) [upload=19509]: Uploading segments... Aug 9 12:06:20.422028174 (18h40m14.853108305s) [upload=19509]: Got next segment. Inline: false Aug 9 12:06:20.422073501 (18h40m14.853153622s) [upload=19509]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 9 12:06:20.422077549 (18h40m14.85315768s) [upload=19509]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 9 12:06:20.634384816 (18h40m15.065464937s) [upload=19509]: Next segment err: metaclient: Too Many Requests Aug 9 12:06:20.634410485 (18h40m15.065490606s) [upload=19509]: Done uploading segments... Aug 9 12:06:20.634426876 (18h40m15.065507007s) [upload=19509]: Upload finished. err: metaclient: Too Many Requests Duration: 5.715073369s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 4 out of 60 is failed (threshold=3) [.................................X............X.X........X..] duration: 5.766523945s FAIL (2025-08-09 12:08:25 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: too many requests ======LOGS====== Aug 9 12:06:20.421719324 (18h40m14.852799455s) [upload=19509]: Starting upload Aug 9 12:06:20.421777855 (18h40m14.852857987s) [upload=19509]: Done starting upload Aug 9 12:06:20.421895961 (18h40m14.852976092s) [upload=19509]: Uploading segments... Aug 9 12:06:20.422028174 (18h40m14.853108305s) [upload=19509]: Got next segment. Inline: false Aug 9 12:06:20.422073501 (18h40m14.853153622s) [upload=19509]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 9 12:06:20.422077549 (18h40m14.85315768s) [upload=19509]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 9 12:06:20.634384816 (18h40m15.065464937s) [upload=19509]: Next segment err: metaclient: Too Many Requests Aug 9 12:06:20.634410485 (18h40m15.065490606s) [upload=19509]: Done uploading segments... Aug 9 12:06:20.634426876 (18h40m15.065507007s) [upload=19509]: Upload finished. err: metaclient: Too Many Requests Duration: 5.766545607s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................X.........................................] duration: 5.699326394s FAIL (2025-08-09 12:08:14 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 6) ======LOGS====== ADHZNIB6LBP5VQHFEGSYM3YNNZWNM2N4UU7JTCQ, seg_pos={0 0}, upload=18998]: Uploading piece... Aug 9 11:37:25.005421531 (18h11m19.436501662s) [noise=true , address=45.91.116.4:4940 , piece_id=M33ZGUCEF23HDGQZE5F5KLWQ43NHC7E7LKOVVGQ6P6TM2HFGB4WQ, seg_pos={0 0}, upload=18998]: Uploading piece... Aug 9 11:37:25.005614329 (18h11m19.43669446s) [noise=true , address=45.91.116.8:4047 , piece_id=B27B2QKZXICP4DHTDTCYSZZ7RN2PPCMKSRQBKRK5JJ4BE4RNWQAQ, seg_pos={0 0}, upload=18998]: Uploading piece... Aug 9 11:37:25.005727536 (18h11m19.436807667s) [noise=true , address=45.91.116.12:4225 , piece_id=VFK2SMVTBT76D7YLQWGRA6TF47JD3B7J7F5NFR4E2MQVNGNHYTAQ, seg_pos={0 0}, upload=18998]: Uploading piece... Aug 9 11:37:25.006281517 (18h11m19.437361648s) [noise=true , address=104.36.82.7:4909 , piece_id=OGG5ZCXSLIM5YKMM7K6KLKLWGOTX376DUTDT3567OB3AJIBPBKDQ, seg_pos={0 0}, upload=18998]: Uploading piece... Aug 9 11:37:25.006632558 (18h11m19.437712679s) [noise=true , address=45.91.116.13:4155 , piece_id=K4CGD4OCR2B6ZWAZHB5MDBECVJB3AC2HDXS2TUVAVGIWQYJEARLA, seg_pos={0 0}, upload=18998]: Uploading piece... Aug 9 11:37:25.006996264 (18h11m19.438076395s) [noise=true , address=77.90.31.18:4806 , piece_id=QCURQYEJL22LBT2YFBR62RIHH7VNSGL7GERP66GKHXK3JP5WLVZQ, seg_pos={0 0}, upload=18998]: Uploading piece... Aug 9 11:37:25.007245231 (18h11m19.438325362s) [noise=true , address=104.36.82.3:4765 , piece_id=HBLTTPZYK5ZXNG2XV53QVJA3NPX3YWKEAAAFVMDXUO6BLR3QKCWQ, seg_pos={0 0}, upload=18998]: Uploading piece... Aug 9 11:37:25.007475481 (18h11m19.438555612s) [noise=true , address=45.91.116.6:4973 , piece_id=MHDNXTVKFMLRKGS7JIPL4U5VQMBXGP4EYM7JEGPSK4U3GIJ5JEHQ, seg_pos={0 0}, upload=18998]: Uploading piece... Aug 9 11:37:25.007873984 (18h11m19.438954115s) [noise=true , address=45.91.116.7:4010 , piece_id=5G3O552GD5XO7SNIHST623QJBWJF2C7B6EN7ISGHHY3FYV6QL2DQ, seg_pos={0 0}, upload=18998]: Uploading piece... Aug 9 11:37:26.733124951 (18h11m21.164205082s) [upload=18998]: Next returned nil segment Aug 9 11:37:26.733142194 (18h11m21.164222325s) [upload=18998]: Waiting for error group managing segments... Aug 9 11:37:27.100757697 (18h11m21.531837848s) [noise=true , address=104.36.82.14:4222 , piece_id=Z4X6RJ5HBAYGUNKT6DXWSS2TEG2OUFPMH3NWYGA7D26XIFH46UAA, seg_pos={0 0}, upload=18998]: Done uploading piece. err: <nil> Aug 9 11:37:27.106781913 (18h11m21.537862064s) [noise=true , address=104.36.82.7:4912 , piece_id=MRA5X6N6MDQ2TADHZNIB6LBP5VQHFEGSYM3YNNZWNM2N4UU7JTCQ, seg_pos={0 0}, upload=18998]: Done uploading piece. err: <nil> Aug 9 11:37:27.136679105 (18h11m21.567759236s) [noise=true , address=104.36.82.5:4838 , piece_id=T3WCWCLB4WAXQY3S6VCFHLKYK6IY2Q6P2776CLGYS4QOFTXMYVEQ, seg_pos={0 0}, upload=18998]: Done uploading piece. err: <nil> Aug 9 11:37:27.139126447 (18h11m21.570206578s) [noise=true , address=104.36.82.3:4765 , piece_id=HBLTTPZYK5ZXNG2XV53QVJA3NPX3YWKEAAAFVMDXUO6BLR3QKCWQ, seg_pos={0 0}, upload=18998]: Done uploading piece. err: <nil> Aug 9 11:37:27.183514474 (18h11m21.614594606s) [noise=true , address=104.36.82.7:4909 , piece_id=OGG5ZCXSLIM5YKMM7K6KLKLWGOTX376DUTDT3567OB3AJIBPBKDQ, seg_pos={0 0}, upload=18998]: Done uploading piece. err: <nil> Aug 9 11:37:53.489259032 (18h11m47.920339163s) [noise=true , address=45.91.116.13:4155 , piece_id=K4CGD4OCR2B6ZWAZHB5MDBECVJB3AC2HDXS2TUVAVGIWQYJEARLA, seg_pos={0 0}, upload=18998]: Done uploading piece. err: <nil> Aug 9 11:37:54.124525138 (18h11m48.555605259s) [noise=true , address=45.91.116.7:4010 , piece_id=5G3O552GD5XO7SNIHST623QJBWJF2C7B6EN7ISGHHY3FYV6QL2DQ, seg_pos={0 0}, upload=18998]: Done uploading piece. err: ecclient: upload failed (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w, address:45.91.116.7:4010): protocol: write tcp 188.166.225.170:54960->45.91.116.7:4010: use of closed network connection; write tcp 188.166.225.170:54960->45.91.116.7:4010: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54960->45.91.116.7:4010: use of closed network connection Aug 9 11:37:54.124581035 (18h11m48.555661166s) [noise=true , address=77.90.31.10:4520 , piece_id=YRBBWM54FGU3B74JQ7EOAO7VSGVFXAVT4SNZ6KN3IQFIHIEVV5HA, seg_pos={0 0}, upload=18998]: Done uploading piece. err: ecclient: upload failed (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp, address:77.90.31.10:4520): protocol: write tcp 188.166.225.170:39560->77.90.31.10:4520: use of closed network connection; write tcp 188.166.225.170:39560->77.90.31.10:4520: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39560->77.90.31.10:4520: use of closed network connection Aug 9 11:37:54.124615480 (18h11m48.555695652s) [noise=true , address=45.91.116.12:4225 , piece_id=VFK2SMVTBT76D7YLQWGRA6TF47JD3B7J7F5NFR4E2MQVNGNHYTAQ, seg_pos={0 0}, upload=18998]: Done uploading piece. err: ecclient: upload failed (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy, address:45.91.116.12:4225): protocol: write tcp 188.166.225.170:38054->45.91.116.12:4225: use of closed network connection; write tcp 188.166.225.170:38054->45.91.116.12:4225: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38054->45.91.116.12:4225: use of closed network connection Aug 9 11:37:54.124643875 (18h11m48.555724016s) [noise=true , address=45.91.116.4:4940 , piece_id=M33ZGUCEF23HDGQZE5F5KLWQ43NHC7E7LKOVVGQ6P6TM2HFGB4WQ, seg_pos={0 0}, upload=18998]: Done uploading piece. err: ecclient: upload failed (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe, address:45.91.116.4:4940): protocol: write tcp 188.166.225.170:40630->45.91.116.4:4940: use of closed network connection; write tcp 188.166.225.170:40630->45.91.116.4:4940: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40630->45.91.116.4:4940: use of closed network connection Aug 9 11:37:54.124693159 (18h11m48.55577335s) [noise=true , address=45.91.116.14:4189 , piece_id=XVQMTYHB6H3IKSERYT2BEB4NP42KV6FRE6J2MKKU5WEOCVBHYIWA, seg_pos={0 0}, upload=18998]: Done uploading piece. err: ecclient: upload failed (node:1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N, address:45.91.116.14:4189): protocol: write tcp 188.166.225.170:37650->45.91.116.14:4189: use of closed network connection; write tcp 188.166.225.170:37650->45.91.116.14:4189: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37650->45.91.116.14:4189: use of closed network connection Aug 9 11:37:54.124771740 (18h11m48.555851871s) [noise=true , address=77.90.31.18:4806 , piece_id=QCURQYEJL22LBT2YFBR62RIHH7VNSGL7GERP66GKHXK3JP5WLVZQ, seg_pos={0 0}, upload=18998]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: write tcp 188.166.225.170:55536->77.90.31.18:4806: use of closed network connection; write tcp 188.166.225.170:55536->77.90.31.18:4806: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55536->77.90.31.18:4806: use of closed network connection Aug 9 11:37:54.124817217 (18h11m48.555897348s) [noise=true , address=77.90.31.15:4697 , piece_id=Q26SRGZO2YBJWPDIG4U422EICE6XEXPLHFHXLKFUBKMMJAZ4W7UQ, seg_pos={0 0}, upload=18998]: Done uploading piece. err: ecclient: upload failed (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, address:77.90.31.15:4697): protocol: write tcp 188.166.225.170:49618->77.90.31.15:4697: use of closed network connection; write tcp 188.166.225.170:49618->77.90.31.15:4697: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49618->77.90.31.15:4697: use of closed network connection Aug 9 11:37:54.124833387 (18h11m48.555913559s) [noise=true , address=45.91.116.6:4973 , piece_id=MHDNXTVKFMLRKGS7JIPL4U5VQMBXGP4EYM7JEGPSK4U3GIJ5JEHQ, seg_pos={0 0}, upload=18998]: Done uploading piece. err: ecclient: upload failed (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, address:45.91.116.6:4973): protocol: write tcp 188.166.225.170:59438->45.91.116.6:4973: use of closed network connection; write tcp 188.166.225.170:59438->45.91.116.6:4973: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59438->45.91.116.6:4973: use of closed network connection Aug 9 11:37:54.124840752 (18h11m48.555920883s) [noise=true , address=77.90.31.17:4772 , piece_id=MGZPHOODRIOTGABQJHNIP5FGF4XN6A723ZHUBKI6KNAJDKFWWOJA, seg_pos={0 0}, upload=18998]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: write tcp 188.166.225.170:42704->77.90.31.17:4772: use of closed network connection; write tcp 188.166.225.170:42704->77.90.31.17:4772: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42704->77.90.31.17:4772: use of closed network connection Aug 9 11:37:54.124845251 (18h11m48.555925382s) [noise=true , address=45.91.116.8:4047 , piece_id=B27B2QKZXICP4DHTDTCYSZZ7RN2PPCMKSRQBKRK5JJ4BE4RNWQAQ, seg_pos={0 0}, upload=18998]: Done uploading piece. err: ecclient: upload failed (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN, address:45.91.116.8:4047): protocol: write tcp 188.166.225.170:40794->45.91.116.8:4047: use of closed network connection; write tcp 188.166.225.170:40794->45.91.116.8:4047: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40794->45.91.116.8:4047: use of closed network connection Aug 9 11:37:54.124866321 (18h11m48.555946492s) [seg_pos={0 0}, upload=18998]: Done waiting for segment. successful: 6 optimal: 13 errs: <nil> Aug 9 11:37:54.353990811 (18h11m48.785070942s) [upload=18998]: Done uploading segments... Aug 9 11:37:54.354028673 (18h11m48.785108804s) [upload=18998]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 6) Duration: 5.699353485s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 3 out of 60 is failed (threshold=3) [...................X..................X.........X...........] duration: 1.851988606s FAIL (2025-08-09 12:08:14 UTC) orbiter warning: operation: ListObjects ======ERROR====== unable to find key in bucket: sj://uptime-checks-placement-20/orbiter_us_0/1K ======LOGS====== Aug 9 11:59:31.819436107 (18h33m26.250516238s) [upload=19388]: Starting upload Aug 9 11:59:31.819492355 (18h33m26.250572516s) [upload=19388]: Done starting upload Aug 9 11:59:31.819563080 (18h33m26.250643211s) [upload=19388]: Uploading segments... Aug 9 11:59:31.819616853 (18h33m26.250696984s) [upload=19388]: Got next segment. Inline: true Aug 9 11:59:31.819621552 (18h33m26.250701683s) [upload=19388]: Waiting for error group managing segments... Aug 9 11:59:31.819660466 (18h33m26.250740597s) [upload=19388]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 9 11:59:31.819665206 (18h33m26.250745327s) [upload=19388]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 9 11:59:31.819667851 (18h33m26.250747972s) [upload=19388]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 9 11:59:32.242597809 (18h33m26.67367794s) [upload=19388]: Done uploading segments... Aug 9 11:59:32.242615142 (18h33m26.673695263s) [upload=19388]: Upload finished. err: <nil> Duration:* 1.852023103s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 3 out of 60 is failed (threshold=3) [....................X..................X.........X..........] duration: 1.85969477s FAIL (2025-08-09 12:07:52 UTC) orbiter warning: operation: ListObjects ======ERROR====== unable to find key in bucket: sj://uptime-checks-placement-20/orbiter_us_0/1K ======LOGS====== Aug 9 11:59:31.819436107 (18h33m26.250516238s) [upload=19388]: Starting upload Aug 9 11:59:31.819492355 (18h33m26.250572516s) [upload=19388]: Done starting upload Aug 9 11:59:31.819563080 (18h33m26.250643211s) [upload=19388]: Uploading segments... Aug 9 11:59:31.819616853 (18h33m26.250696984s) [upload=19388]: Got next segment. Inline: true Aug 9 11:59:31.819621552 (18h33m26.250701683s) [upload=19388]: Waiting for error group managing segments... Aug 9 11:59:31.819660466 (18h33m26.250740597s) [upload=19388]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 9 11:59:31.819665206 (18h33m26.250745327s) [upload=19388]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 9 11:59:31.819667851 (18h33m26.250747972s) [upload=19388]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 9 11:59:32.242597809 (18h33m26.67367794s) [upload=19388]: Done uploading segments... Aug 9 11:59:32.242615142 (18h33m26.673695263s) [upload=19388]: Upload finished. err: <nil> Duration:* 1.859718497s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................X........................................] duration: 5.768139802s FAIL (2025-08-09 12:07:31 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 6) ======LOGS====== ADHZNIB6LBP5VQHFEGSYM3YNNZWNM2N4UU7JTCQ, seg_pos={0 0}, upload=18998]: Uploading piece... Aug 9 11:37:25.005421531 (18h11m19.436501662s) [noise=true , address=45.91.116.4:4940 , piece_id=M33ZGUCEF23HDGQZE5F5KLWQ43NHC7E7LKOVVGQ6P6TM2HFGB4WQ, seg_pos={0 0}, upload=18998]: Uploading piece... Aug 9 11:37:25.005614329 (18h11m19.43669446s) [noise=true , address=45.91.116.8:4047 , piece_id=B27B2QKZXICP4DHTDTCYSZZ7RN2PPCMKSRQBKRK5JJ4BE4RNWQAQ, seg_pos={0 0}, upload=18998]: Uploading piece... Aug 9 11:37:25.005727536 (18h11m19.436807667s) [noise=true , address=45.91.116.12:4225 , piece_id=VFK2SMVTBT76D7YLQWGRA6TF47JD3B7J7F5NFR4E2MQVNGNHYTAQ, seg_pos={0 0}, upload=18998]: Uploading piece... Aug 9 11:37:25.006281517 (18h11m19.437361648s) [noise=true , address=104.36.82.7:4909 , piece_id=OGG5ZCXSLIM5YKMM7K6KLKLWGOTX376DUTDT3567OB3AJIBPBKDQ, seg_pos={0 0}, upload=18998]: Uploading piece... Aug 9 11:37:25.006632558 (18h11m19.437712679s) [noise=true , address=45.91.116.13:4155 , piece_id=K4CGD4OCR2B6ZWAZHB5MDBECVJB3AC2HDXS2TUVAVGIWQYJEARLA, seg_pos={0 0}, upload=18998]: Uploading piece... Aug 9 11:37:25.006996264 (18h11m19.438076395s) [noise=true , address=77.90.31.18:4806 , piece_id=QCURQYEJL22LBT2YFBR62RIHH7VNSGL7GERP66GKHXK3JP5WLVZQ, seg_pos={0 0}, upload=18998]: Uploading piece... Aug 9 11:37:25.007245231 (18h11m19.438325362s) [noise=true , address=104.36.82.3:4765 , piece_id=HBLTTPZYK5ZXNG2XV53QVJA3NPX3YWKEAAAFVMDXUO6BLR3QKCWQ, seg_pos={0 0}, upload=18998]: Uploading piece... Aug 9 11:37:25.007475481 (18h11m19.438555612s) [noise=true , address=45.91.116.6:4973 , piece_id=MHDNXTVKFMLRKGS7JIPL4U5VQMBXGP4EYM7JEGPSK4U3GIJ5JEHQ, seg_pos={0 0}, upload=18998]: Uploading piece... Aug 9 11:37:25.007873984 (18h11m19.438954115s) [noise=true , address=45.91.116.7:4010 , piece_id=5G3O552GD5XO7SNIHST623QJBWJF2C7B6EN7ISGHHY3FYV6QL2DQ, seg_pos={0 0}, upload=18998]: Uploading piece... Aug 9 11:37:26.733124951 (18h11m21.164205082s) [upload=18998]: Next returned nil segment Aug 9 11:37:26.733142194 (18h11m21.164222325s) [upload=18998]: Waiting for error group managing segments... Aug 9 11:37:27.100757697 (18h11m21.531837848s) [noise=true , address=104.36.82.14:4222 , piece_id=Z4X6RJ5HBAYGUNKT6DXWSS2TEG2OUFPMH3NWYGA7D26XIFH46UAA, seg_pos={0 0}, upload=18998]: Done uploading piece. err: <nil> Aug 9 11:37:27.106781913 (18h11m21.537862064s) [noise=true , address=104.36.82.7:4912 , piece_id=MRA5X6N6MDQ2TADHZNIB6LBP5VQHFEGSYM3YNNZWNM2N4UU7JTCQ, seg_pos={0 0}, upload=18998]: Done uploading piece. err: <nil> Aug 9 11:37:27.136679105 (18h11m21.567759236s) [noise=true , address=104.36.82.5:4838 , piece_id=T3WCWCLB4WAXQY3S6VCFHLKYK6IY2Q6P2776CLGYS4QOFTXMYVEQ, seg_pos={0 0}, upload=18998]: Done uploading piece. err: <nil> Aug 9 11:37:27.139126447 (18h11m21.570206578s) [noise=true , address=104.36.82.3:4765 , piece_id=HBLTTPZYK5ZXNG2XV53QVJA3NPX3YWKEAAAFVMDXUO6BLR3QKCWQ, seg_pos={0 0}, upload=18998]: Done uploading piece. err: <nil> Aug 9 11:37:27.183514474 (18h11m21.614594606s) [noise=true , address=104.36.82.7:4909 , piece_id=OGG5ZCXSLIM5YKMM7K6KLKLWGOTX376DUTDT3567OB3AJIBPBKDQ, seg_pos={0 0}, upload=18998]: Done uploading piece. err: <nil> Aug 9 11:37:53.489259032 (18h11m47.920339163s) [noise=true , address=45.91.116.13:4155 , piece_id=K4CGD4OCR2B6ZWAZHB5MDBECVJB3AC2HDXS2TUVAVGIWQYJEARLA, seg_pos={0 0}, upload=18998]: Done uploading piece. err: <nil> Aug 9 11:37:54.124525138 (18h11m48.555605259s) [noise=true , address=45.91.116.7:4010 , piece_id=5G3O552GD5XO7SNIHST623QJBWJF2C7B6EN7ISGHHY3FYV6QL2DQ, seg_pos={0 0}, upload=18998]: Done uploading piece. err: ecclient: upload failed (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w, address:45.91.116.7:4010): protocol: write tcp 188.166.225.170:54960->45.91.116.7:4010: use of closed network connection; write tcp 188.166.225.170:54960->45.91.116.7:4010: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54960->45.91.116.7:4010: use of closed network connection Aug 9 11:37:54.124581035 (18h11m48.555661166s) [noise=true , address=77.90.31.10:4520 , piece_id=YRBBWM54FGU3B74JQ7EOAO7VSGVFXAVT4SNZ6KN3IQFIHIEVV5HA, seg_pos={0 0}, upload=18998]: Done uploading piece. err: ecclient: upload failed (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp, address:77.90.31.10:4520): protocol: write tcp 188.166.225.170:39560->77.90.31.10:4520: use of closed network connection; write tcp 188.166.225.170:39560->77.90.31.10:4520: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39560->77.90.31.10:4520: use of closed network connection Aug 9 11:37:54.124615480 (18h11m48.555695652s) [noise=true , address=45.91.116.12:4225 , piece_id=VFK2SMVTBT76D7YLQWGRA6TF47JD3B7J7F5NFR4E2MQVNGNHYTAQ, seg_pos={0 0}, upload=18998]: Done uploading piece. err: ecclient: upload failed (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy, address:45.91.116.12:4225): protocol: write tcp 188.166.225.170:38054->45.91.116.12:4225: use of closed network connection; write tcp 188.166.225.170:38054->45.91.116.12:4225: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38054->45.91.116.12:4225: use of closed network connection Aug 9 11:37:54.124643875 (18h11m48.555724016s) [noise=true , address=45.91.116.4:4940 , piece_id=M33ZGUCEF23HDGQZE5F5KLWQ43NHC7E7LKOVVGQ6P6TM2HFGB4WQ, seg_pos={0 0}, upload=18998]: Done uploading piece. err: ecclient: upload failed (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe, address:45.91.116.4:4940): protocol: write tcp 188.166.225.170:40630->45.91.116.4:4940: use of closed network connection; write tcp 188.166.225.170:40630->45.91.116.4:4940: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40630->45.91.116.4:4940: use of closed network connection Aug 9 11:37:54.124693159 (18h11m48.55577335s) [noise=true , address=45.91.116.14:4189 , piece_id=XVQMTYHB6H3IKSERYT2BEB4NP42KV6FRE6J2MKKU5WEOCVBHYIWA, seg_pos={0 0}, upload=18998]: Done uploading piece. err: ecclient: upload failed (node:1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N, address:45.91.116.14:4189): protocol: write tcp 188.166.225.170:37650->45.91.116.14:4189: use of closed network connection; write tcp 188.166.225.170:37650->45.91.116.14:4189: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37650->45.91.116.14:4189: use of closed network connection Aug 9 11:37:54.124771740 (18h11m48.555851871s) [noise=true , address=77.90.31.18:4806 , piece_id=QCURQYEJL22LBT2YFBR62RIHH7VNSGL7GERP66GKHXK3JP5WLVZQ, seg_pos={0 0}, upload=18998]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: write tcp 188.166.225.170:55536->77.90.31.18:4806: use of closed network connection; write tcp 188.166.225.170:55536->77.90.31.18:4806: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55536->77.90.31.18:4806: use of closed network connection Aug 9 11:37:54.124817217 (18h11m48.555897348s) [noise=true , address=77.90.31.15:4697 , piece_id=Q26SRGZO2YBJWPDIG4U422EICE6XEXPLHFHXLKFUBKMMJAZ4W7UQ, seg_pos={0 0}, upload=18998]: Done uploading piece. err: ecclient: upload failed (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, address:77.90.31.15:4697): protocol: write tcp 188.166.225.170:49618->77.90.31.15:4697: use of closed network connection; write tcp 188.166.225.170:49618->77.90.31.15:4697: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49618->77.90.31.15:4697: use of closed network connection Aug 9 11:37:54.124833387 (18h11m48.555913559s) [noise=true , address=45.91.116.6:4973 , piece_id=MHDNXTVKFMLRKGS7JIPL4U5VQMBXGP4EYM7JEGPSK4U3GIJ5JEHQ, seg_pos={0 0}, upload=18998]: Done uploading piece. err: ecclient: upload failed (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, address:45.91.116.6:4973): protocol: write tcp 188.166.225.170:59438->45.91.116.6:4973: use of closed network connection; write tcp 188.166.225.170:59438->45.91.116.6:4973: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59438->45.91.116.6:4973: use of closed network connection Aug 9 11:37:54.124840752 (18h11m48.555920883s) [noise=true , address=77.90.31.17:4772 , piece_id=MGZPHOODRIOTGABQJHNIP5FGF4XN6A723ZHUBKI6KNAJDKFWWOJA, seg_pos={0 0}, upload=18998]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: write tcp 188.166.225.170:42704->77.90.31.17:4772: use of closed network connection; write tcp 188.166.225.170:42704->77.90.31.17:4772: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42704->77.90.31.17:4772: use of closed network connection Aug 9 11:37:54.124845251 (18h11m48.555925382s) [noise=true , address=45.91.116.8:4047 , piece_id=B27B2QKZXICP4DHTDTCYSZZ7RN2PPCMKSRQBKRK5JJ4BE4RNWQAQ, seg_pos={0 0}, upload=18998]: Done uploading piece. err: ecclient: upload failed (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN, address:45.91.116.8:4047): protocol: write tcp 188.166.225.170:40794->45.91.116.8:4047: use of closed network connection; write tcp 188.166.225.170:40794->45.91.116.8:4047: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40794->45.91.116.8:4047: use of closed network connection Aug 9 11:37:54.124866321 (18h11m48.555946492s) [seg_pos={0 0}, upload=18998]: Done waiting for segment. successful: 6 optimal: 13 errs: <nil> Aug 9 11:37:54.353990811 (18h11m48.785070942s) [upload=18998]: Done uploading segments... Aug 9 11:37:54.354028673 (18h11m48.785108804s) [upload=18998]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 6) Duration: 5.768162525s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 4 out of 60 is failed (threshold=3) [..................................X............X.X........X.] duration: 5.674832779s FAIL (2025-08-09 12:07:22 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: too many requests ======LOGS====== Aug 9 12:06:20.421719324 (18h40m14.852799455s) [upload=19509]: Starting upload Aug 9 12:06:20.421777855 (18h40m14.852857987s) [upload=19509]: Done starting upload Aug 9 12:06:20.421895961 (18h40m14.852976092s) [upload=19509]: Uploading segments... Aug 9 12:06:20.422028174 (18h40m14.853108305s) [upload=19509]: Got next segment. Inline: false Aug 9 12:06:20.422073501 (18h40m14.853153622s) [upload=19509]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 9 12:06:20.422077549 (18h40m14.85315768s) [upload=19509]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 9 12:06:20.634384816 (18h40m15.065464937s) [upload=19509]: Next segment err: metaclient: Too Many Requests Aug 9 12:06:20.634410485 (18h40m15.065490606s) [upload=19509]: Done uploading segments... Aug 9 12:06:20.634426876 (18h40m15.065507007s) [upload=19509]: Upload finished. err: metaclient: Too Many Requests Duration: 5.674858999s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 3 out of 60 is failed (threshold=3) [.....................X..................X.........X.........] duration: 2.114386714s FAIL (2025-08-09 12:06:57 UTC) orbiter warning: operation: ListObjects ======ERROR====== unable to find key in bucket: sj://uptime-checks-placement-20/orbiter_us_0/1K ======LOGS====== Aug 9 11:59:31.819436107 (18h33m26.250516238s) [upload=19388]: Starting upload Aug 9 11:59:31.819492355 (18h33m26.250572516s) [upload=19388]: Done starting upload Aug 9 11:59:31.819563080 (18h33m26.250643211s) [upload=19388]: Uploading segments... Aug 9 11:59:31.819616853 (18h33m26.250696984s) [upload=19388]: Got next segment. Inline: true Aug 9 11:59:31.819621552 (18h33m26.250701683s) [upload=19388]: Waiting for error group managing segments... Aug 9 11:59:31.819660466 (18h33m26.250740597s) [upload=19388]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 9 11:59:31.819665206 (18h33m26.250745327s) [upload=19388]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 9 11:59:31.819667851 (18h33m26.250747972s) [upload=19388]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 9 11:59:32.242597809 (18h33m26.67367794s) [upload=19388]: Done uploading segments... Aug 9 11:59:32.242615142 (18h33m26.673695263s) [upload=19388]: Upload finished. err: <nil> Duration:* 2.114411962s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................X.......................................] duration: 5.906786754s FAIL (2025-08-09 12:06:42 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 6) ======LOGS====== ADHZNIB6LBP5VQHFEGSYM3YNNZWNM2N4UU7JTCQ, seg_pos={0 0}, upload=18998]: Uploading piece... Aug 9 11:37:25.005421531 (18h11m19.436501662s) [noise=true , address=45.91.116.4:4940 , piece_id=M33ZGUCEF23HDGQZE5F5KLWQ43NHC7E7LKOVVGQ6P6TM2HFGB4WQ, seg_pos={0 0}, upload=18998]: Uploading piece... Aug 9 11:37:25.005614329 (18h11m19.43669446s) [noise=true , address=45.91.116.8:4047 , piece_id=B27B2QKZXICP4DHTDTCYSZZ7RN2PPCMKSRQBKRK5JJ4BE4RNWQAQ, seg_pos={0 0}, upload=18998]: Uploading piece... Aug 9 11:37:25.005727536 (18h11m19.436807667s) [noise=true , address=45.91.116.12:4225 , piece_id=VFK2SMVTBT76D7YLQWGRA6TF47JD3B7J7F5NFR4E2MQVNGNHYTAQ, seg_pos={0 0}, upload=18998]: Uploading piece... Aug 9 11:37:25.006281517 (18h11m19.437361648s) [noise=true , address=104.36.82.7:4909 , piece_id=OGG5ZCXSLIM5YKMM7K6KLKLWGOTX376DUTDT3567OB3AJIBPBKDQ, seg_pos={0 0}, upload=18998]: Uploading piece... Aug 9 11:37:25.006632558 (18h11m19.437712679s) [noise=true , address=45.91.116.13:4155 , piece_id=K4CGD4OCR2B6ZWAZHB5MDBECVJB3AC2HDXS2TUVAVGIWQYJEARLA, seg_pos={0 0}, upload=18998]: Uploading piece... Aug 9 11:37:25.006996264 (18h11m19.438076395s) [noise=true , address=77.90.31.18:4806 , piece_id=QCURQYEJL22LBT2YFBR62RIHH7VNSGL7GERP66GKHXK3JP5WLVZQ, seg_pos={0 0}, upload=18998]: Uploading piece... Aug 9 11:37:25.007245231 (18h11m19.438325362s) [noise=true , address=104.36.82.3:4765 , piece_id=HBLTTPZYK5ZXNG2XV53QVJA3NPX3YWKEAAAFVMDXUO6BLR3QKCWQ, seg_pos={0 0}, upload=18998]: Uploading piece... Aug 9 11:37:25.007475481 (18h11m19.438555612s) [noise=true , address=45.91.116.6:4973 , piece_id=MHDNXTVKFMLRKGS7JIPL4U5VQMBXGP4EYM7JEGPSK4U3GIJ5JEHQ, seg_pos={0 0}, upload=18998]: Uploading piece... Aug 9 11:37:25.007873984 (18h11m19.438954115s) [noise=true , address=45.91.116.7:4010 , piece_id=5G3O552GD5XO7SNIHST623QJBWJF2C7B6EN7ISGHHY3FYV6QL2DQ, seg_pos={0 0}, upload=18998]: Uploading piece... Aug 9 11:37:26.733124951 (18h11m21.164205082s) [upload=18998]: Next returned nil segment Aug 9 11:37:26.733142194 (18h11m21.164222325s) [upload=18998]: Waiting for error group managing segments... Aug 9 11:37:27.100757697 (18h11m21.531837848s) [noise=true , address=104.36.82.14:4222 , piece_id=Z4X6RJ5HBAYGUNKT6DXWSS2TEG2OUFPMH3NWYGA7D26XIFH46UAA, seg_pos={0 0}, upload=18998]: Done uploading piece. err: <nil> Aug 9 11:37:27.106781913 (18h11m21.537862064s) [noise=true , address=104.36.82.7:4912 , piece_id=MRA5X6N6MDQ2TADHZNIB6LBP5VQHFEGSYM3YNNZWNM2N4UU7JTCQ, seg_pos={0 0}, upload=18998]: Done uploading piece. err: <nil> Aug 9 11:37:27.136679105 (18h11m21.567759236s) [noise=true , address=104.36.82.5:4838 , piece_id=T3WCWCLB4WAXQY3S6VCFHLKYK6IY2Q6P2776CLGYS4QOFTXMYVEQ, seg_pos={0 0}, upload=18998]: Done uploading piece. err: <nil> Aug 9 11:37:27.139126447 (18h11m21.570206578s) [noise=true , address=104.36.82.3:4765 , piece_id=HBLTTPZYK5ZXNG2XV53QVJA3NPX3YWKEAAAFVMDXUO6BLR3QKCWQ, seg_pos={0 0}, upload=18998]: Done uploading piece. err: <nil> Aug 9 11:37:27.183514474 (18h11m21.614594606s) [noise=true , address=104.36.82.7:4909 , piece_id=OGG5ZCXSLIM5YKMM7K6KLKLWGOTX376DUTDT3567OB3AJIBPBKDQ, seg_pos={0 0}, upload=18998]: Done uploading piece. err: <nil> Aug 9 11:37:53.489259032 (18h11m47.920339163s) [noise=true , address=45.91.116.13:4155 , piece_id=K4CGD4OCR2B6ZWAZHB5MDBECVJB3AC2HDXS2TUVAVGIWQYJEARLA, seg_pos={0 0}, upload=18998]: Done uploading piece. err: <nil> Aug 9 11:37:54.124525138 (18h11m48.555605259s) [noise=true , address=45.91.116.7:4010 , piece_id=5G3O552GD5XO7SNIHST623QJBWJF2C7B6EN7ISGHHY3FYV6QL2DQ, seg_pos={0 0}, upload=18998]: Done uploading piece. err: ecclient: upload failed (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w, address:45.91.116.7:4010): protocol: write tcp 188.166.225.170:54960->45.91.116.7:4010: use of closed network connection; write tcp 188.166.225.170:54960->45.91.116.7:4010: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54960->45.91.116.7:4010: use of closed network connection Aug 9 11:37:54.124581035 (18h11m48.555661166s) [noise=true , address=77.90.31.10:4520 , piece_id=YRBBWM54FGU3B74JQ7EOAO7VSGVFXAVT4SNZ6KN3IQFIHIEVV5HA, seg_pos={0 0}, upload=18998]: Done uploading piece. err: ecclient: upload failed (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp, address:77.90.31.10:4520): protocol: write tcp 188.166.225.170:39560->77.90.31.10:4520: use of closed network connection; write tcp 188.166.225.170:39560->77.90.31.10:4520: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39560->77.90.31.10:4520: use of closed network connection Aug 9 11:37:54.124615480 (18h11m48.555695652s) [noise=true , address=45.91.116.12:4225 , piece_id=VFK2SMVTBT76D7YLQWGRA6TF47JD3B7J7F5NFR4E2MQVNGNHYTAQ, seg_pos={0 0}, upload=18998]: Done uploading piece. err: ecclient: upload failed (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy, address:45.91.116.12:4225): protocol: write tcp 188.166.225.170:38054->45.91.116.12:4225: use of closed network connection; write tcp 188.166.225.170:38054->45.91.116.12:4225: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38054->45.91.116.12:4225: use of closed network connection Aug 9 11:37:54.124643875 (18h11m48.555724016s) [noise=true , address=45.91.116.4:4940 , piece_id=M33ZGUCEF23HDGQZE5F5KLWQ43NHC7E7LKOVVGQ6P6TM2HFGB4WQ, seg_pos={0 0}, upload=18998]: Done uploading piece. err: ecclient: upload failed (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe, address:45.91.116.4:4940): protocol: write tcp 188.166.225.170:40630->45.91.116.4:4940: use of closed network connection; write tcp 188.166.225.170:40630->45.91.116.4:4940: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40630->45.91.116.4:4940: use of closed network connection Aug 9 11:37:54.124693159 (18h11m48.55577335s) [noise=true , address=45.91.116.14:4189 , piece_id=XVQMTYHB6H3IKSERYT2BEB4NP42KV6FRE6J2MKKU5WEOCVBHYIWA, seg_pos={0 0}, upload=18998]: Done uploading piece. err: ecclient: upload failed (node:1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N, address:45.91.116.14:4189): protocol: write tcp 188.166.225.170:37650->45.91.116.14:4189: use of closed network connection; write tcp 188.166.225.170:37650->45.91.116.14:4189: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37650->45.91.116.14:4189: use of closed network connection Aug 9 11:37:54.124771740 (18h11m48.555851871s) [noise=true , address=77.90.31.18:4806 , piece_id=QCURQYEJL22LBT2YFBR62RIHH7VNSGL7GERP66GKHXK3JP5WLVZQ, seg_pos={0 0}, upload=18998]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: write tcp 188.166.225.170:55536->77.90.31.18:4806: use of closed network connection; write tcp 188.166.225.170:55536->77.90.31.18:4806: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55536->77.90.31.18:4806: use of closed network connection Aug 9 11:37:54.124817217 (18h11m48.555897348s) [noise=true , address=77.90.31.15:4697 , piece_id=Q26SRGZO2YBJWPDIG4U422EICE6XEXPLHFHXLKFUBKMMJAZ4W7UQ, seg_pos={0 0}, upload=18998]: Done uploading piece. err: ecclient: upload failed (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, address:77.90.31.15:4697): protocol: write tcp 188.166.225.170:49618->77.90.31.15:4697: use of closed network connection; write tcp 188.166.225.170:49618->77.90.31.15:4697: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49618->77.90.31.15:4697: use of closed network connection Aug 9 11:37:54.124833387 (18h11m48.555913559s) [noise=true , address=45.91.116.6:4973 , piece_id=MHDNXTVKFMLRKGS7JIPL4U5VQMBXGP4EYM7JEGPSK4U3GIJ5JEHQ, seg_pos={0 0}, upload=18998]: Done uploading piece. err: ecclient: upload failed (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, address:45.91.116.6:4973): protocol: write tcp 188.166.225.170:59438->45.91.116.6:4973: use of closed network connection; write tcp 188.166.225.170:59438->45.91.116.6:4973: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59438->45.91.116.6:4973: use of closed network connection Aug 9 11:37:54.124840752 (18h11m48.555920883s) [noise=true , address=77.90.31.17:4772 , piece_id=MGZPHOODRIOTGABQJHNIP5FGF4XN6A723ZHUBKI6KNAJDKFWWOJA, seg_pos={0 0}, upload=18998]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: write tcp 188.166.225.170:42704->77.90.31.17:4772: use of closed network connection; write tcp 188.166.225.170:42704->77.90.31.17:4772: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42704->77.90.31.17:4772: use of closed network connection Aug 9 11:37:54.124845251 (18h11m48.555925382s) [noise=true , address=45.91.116.8:4047 , piece_id=B27B2QKZXICP4DHTDTCYSZZ7RN2PPCMKSRQBKRK5JJ4BE4RNWQAQ, seg_pos={0 0}, upload=18998]: Done uploading piece. err: ecclient: upload failed (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN, address:45.91.116.8:4047): protocol: write tcp 188.166.225.170:40794->45.91.116.8:4047: use of closed network connection; write tcp 188.166.225.170:40794->45.91.116.8:4047: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40794->45.91.116.8:4047: use of closed network connection Aug 9 11:37:54.124866321 (18h11m48.555946492s) [seg_pos={0 0}, upload=18998]: Done waiting for segment. successful: 6 optimal: 13 errs: <nil> Aug 9 11:37:54.353990811 (18h11m48.785070942s) [upload=18998]: Done uploading segments... Aug 9 11:37:54.354028673 (18h11m48.785108804s) [upload=18998]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 6) Duration: 5.906817393s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 4 out of 60 is failed (threshold=3) [...................................X............X.X........X] duration: 856.217437ms FAIL (2025-08-09 12:06:20 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: too many requests ======LOGS====== Aug 9 12:06:20.421719324 (18h40m14.852799455s) [upload=19509]: Starting upload Aug 9 12:06:20.421777855 (18h40m14.852857987s) [upload=19509]: Done starting upload Aug 9 12:06:20.421895961 (18h40m14.852976092s) [upload=19509]: Uploading segments... Aug 9 12:06:20.422028174 (18h40m14.853108305s) [upload=19509]: Got next segment. Inline: false Aug 9 12:06:20.422073501 (18h40m14.853153622s) [upload=19509]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 9 12:06:20.422077549 (18h40m14.85315768s) [upload=19509]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 9 12:06:20.634384816 (18h40m15.065464937s) [upload=19509]: Next segment err: metaclient: Too Many Requests Aug 9 12:06:20.634410485 (18h40m15.065490606s) [upload=19509]: Done uploading segments... Aug 9 12:06:20.634426876 (18h40m15.065507007s) [upload=19509]: Upload finished. err: metaclient: Too Many Requests Duration: 856.241532ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 3 out of 60 is failed (threshold=3) [......................X..................X.........X........] duration: 1.894092264s FAIL (2025-08-09 12:05:48 UTC) orbiter warning: operation: ListObjects ======ERROR====== unable to find key in bucket: sj://uptime-checks-placement-20/orbiter_us_0/1K ======LOGS====== Aug 9 11:59:31.819436107 (18h33m26.250516238s) [upload=19388]: Starting upload Aug 9 11:59:31.819492355 (18h33m26.250572516s) [upload=19388]: Done starting upload Aug 9 11:59:31.819563080 (18h33m26.250643211s) [upload=19388]: Uploading segments... Aug 9 11:59:31.819616853 (18h33m26.250696984s) [upload=19388]: Got next segment. Inline: true Aug 9 11:59:31.819621552 (18h33m26.250701683s) [upload=19388]: Waiting for error group managing segments... Aug 9 11:59:31.819660466 (18h33m26.250740597s) [upload=19388]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 9 11:59:31.819665206 (18h33m26.250745327s) [upload=19388]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 9 11:59:31.819667851 (18h33m26.250747972s) [upload=19388]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 9 11:59:32.242597809 (18h33m26.67367794s) [upload=19388]: Done uploading segments... Aug 9 11:59:32.242615142 (18h33m26.673695263s) [upload=19388]: Upload finished. err: <nil> Duration:* 1.894119236s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................X......................................] duration: 5.650376636s FAIL (2025-08-09 12:05:41 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 6) ======LOGS====== ADHZNIB6LBP5VQHFEGSYM3YNNZWNM2N4UU7JTCQ, seg_pos={0 0}, upload=18998]: Uploading piece... Aug 9 11:37:25.005421531 (18h11m19.436501662s) [noise=true , address=45.91.116.4:4940 , piece_id=M33ZGUCEF23HDGQZE5F5KLWQ43NHC7E7LKOVVGQ6P6TM2HFGB4WQ, seg_pos={0 0}, upload=18998]: Uploading piece... Aug 9 11:37:25.005614329 (18h11m19.43669446s) [noise=true , address=45.91.116.8:4047 , piece_id=B27B2QKZXICP4DHTDTCYSZZ7RN2PPCMKSRQBKRK5JJ4BE4RNWQAQ, seg_pos={0 0}, upload=18998]: Uploading piece... Aug 9 11:37:25.005727536 (18h11m19.436807667s) [noise=true , address=45.91.116.12:4225 , piece_id=VFK2SMVTBT76D7YLQWGRA6TF47JD3B7J7F5NFR4E2MQVNGNHYTAQ, seg_pos={0 0}, upload=18998]: Uploading piece... Aug 9 11:37:25.006281517 (18h11m19.437361648s) [noise=true , address=104.36.82.7:4909 , piece_id=OGG5ZCXSLIM5YKMM7K6KLKLWGOTX376DUTDT3567OB3AJIBPBKDQ, seg_pos={0 0}, upload=18998]: Uploading piece... Aug 9 11:37:25.006632558 (18h11m19.437712679s) [noise=true , address=45.91.116.13:4155 , piece_id=K4CGD4OCR2B6ZWAZHB5MDBECVJB3AC2HDXS2TUVAVGIWQYJEARLA, seg_pos={0 0}, upload=18998]: Uploading piece... Aug 9 11:37:25.006996264 (18h11m19.438076395s) [noise=true , address=77.90.31.18:4806 , piece_id=QCURQYEJL22LBT2YFBR62RIHH7VNSGL7GERP66GKHXK3JP5WLVZQ, seg_pos={0 0}, upload=18998]: Uploading piece... Aug 9 11:37:25.007245231 (18h11m19.438325362s) [noise=true , address=104.36.82.3:4765 , piece_id=HBLTTPZYK5ZXNG2XV53QVJA3NPX3YWKEAAAFVMDXUO6BLR3QKCWQ, seg_pos={0 0}, upload=18998]: Uploading piece... Aug 9 11:37:25.007475481 (18h11m19.438555612s) [noise=true , address=45.91.116.6:4973 , piece_id=MHDNXTVKFMLRKGS7JIPL4U5VQMBXGP4EYM7JEGPSK4U3GIJ5JEHQ, seg_pos={0 0}, upload=18998]: Uploading piece... Aug 9 11:37:25.007873984 (18h11m19.438954115s) [noise=true , address=45.91.116.7:4010 , piece_id=5G3O552GD5XO7SNIHST623QJBWJF2C7B6EN7ISGHHY3FYV6QL2DQ, seg_pos={0 0}, upload=18998]: Uploading piece... Aug 9 11:37:26.733124951 (18h11m21.164205082s) [upload=18998]: Next returned nil segment Aug 9 11:37:26.733142194 (18h11m21.164222325s) [upload=18998]: Waiting for error group managing segments... Aug 9 11:37:27.100757697 (18h11m21.531837848s) [noise=true , address=104.36.82.14:4222 , piece_id=Z4X6RJ5HBAYGUNKT6DXWSS2TEG2OUFPMH3NWYGA7D26XIFH46UAA, seg_pos={0 0}, upload=18998]: Done uploading piece. err: <nil> Aug 9 11:37:27.106781913 (18h11m21.537862064s) [noise=true , address=104.36.82.7:4912 , piece_id=MRA5X6N6MDQ2TADHZNIB6LBP5VQHFEGSYM3YNNZWNM2N4UU7JTCQ, seg_pos={0 0}, upload=18998]: Done uploading piece. err: <nil> Aug 9 11:37:27.136679105 (18h11m21.567759236s) [noise=true , address=104.36.82.5:4838 , piece_id=T3WCWCLB4WAXQY3S6VCFHLKYK6IY2Q6P2776CLGYS4QOFTXMYVEQ, seg_pos={0 0}, upload=18998]: Done uploading piece. err: <nil> Aug 9 11:37:27.139126447 (18h11m21.570206578s) [noise=true , address=104.36.82.3:4765 , piece_id=HBLTTPZYK5ZXNG2XV53QVJA3NPX3YWKEAAAFVMDXUO6BLR3QKCWQ, seg_pos={0 0}, upload=18998]: Done uploading piece. err: <nil> Aug 9 11:37:27.183514474 (18h11m21.614594606s) [noise=true , address=104.36.82.7:4909 , piece_id=OGG5ZCXSLIM5YKMM7K6KLKLWGOTX376DUTDT3567OB3AJIBPBKDQ, seg_pos={0 0}, upload=18998]: Done uploading piece. err: <nil> Aug 9 11:37:53.489259032 (18h11m47.920339163s) [noise=true , address=45.91.116.13:4155 , piece_id=K4CGD4OCR2B6ZWAZHB5MDBECVJB3AC2HDXS2TUVAVGIWQYJEARLA, seg_pos={0 0}, upload=18998]: Done uploading piece. err: <nil> Aug 9 11:37:54.124525138 (18h11m48.555605259s) [noise=true , address=45.91.116.7:4010 , piece_id=5G3O552GD5XO7SNIHST623QJBWJF2C7B6EN7ISGHHY3FYV6QL2DQ, seg_pos={0 0}, upload=18998]: Done uploading piece. err: ecclient: upload failed (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w, address:45.91.116.7:4010): protocol: write tcp 188.166.225.170:54960->45.91.116.7:4010: use of closed network connection; write tcp 188.166.225.170:54960->45.91.116.7:4010: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54960->45.91.116.7:4010: use of closed network connection Aug 9 11:37:54.124581035 (18h11m48.555661166s) [noise=true , address=77.90.31.10:4520 , piece_id=YRBBWM54FGU3B74JQ7EOAO7VSGVFXAVT4SNZ6KN3IQFIHIEVV5HA, seg_pos={0 0}, upload=18998]: Done uploading piece. err: ecclient: upload failed (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp, address:77.90.31.10:4520): protocol: write tcp 188.166.225.170:39560->77.90.31.10:4520: use of closed network connection; write tcp 188.166.225.170:39560->77.90.31.10:4520: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39560->77.90.31.10:4520: use of closed network connection Aug 9 11:37:54.124615480 (18h11m48.555695652s) [noise=true , address=45.91.116.12:4225 , piece_id=VFK2SMVTBT76D7YLQWGRA6TF47JD3B7J7F5NFR4E2MQVNGNHYTAQ, seg_pos={0 0}, upload=18998]: Done uploading piece. err: ecclient: upload failed (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy, address:45.91.116.12:4225): protocol: write tcp 188.166.225.170:38054->45.91.116.12:4225: use of closed network connection; write tcp 188.166.225.170:38054->45.91.116.12:4225: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38054->45.91.116.12:4225: use of closed network connection Aug 9 11:37:54.124643875 (18h11m48.555724016s) [noise=true , address=45.91.116.4:4940 , piece_id=M33ZGUCEF23HDGQZE5F5KLWQ43NHC7E7LKOVVGQ6P6TM2HFGB4WQ, seg_pos={0 0}, upload=18998]: Done uploading piece. err: ecclient: upload failed (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe, address:45.91.116.4:4940): protocol: write tcp 188.166.225.170:40630->45.91.116.4:4940: use of closed network connection; write tcp 188.166.225.170:40630->45.91.116.4:4940: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40630->45.91.116.4:4940: use of closed network connection Aug 9 11:37:54.124693159 (18h11m48.55577335s) [noise=true , address=45.91.116.14:4189 , piece_id=XVQMTYHB6H3IKSERYT2BEB4NP42KV6FRE6J2MKKU5WEOCVBHYIWA, seg_pos={0 0}, upload=18998]: Done uploading piece. err: ecclient: upload failed (node:1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N, address:45.91.116.14:4189): protocol: write tcp 188.166.225.170:37650->45.91.116.14:4189: use of closed network connection; write tcp 188.166.225.170:37650->45.91.116.14:4189: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37650->45.91.116.14:4189: use of closed network connection Aug 9 11:37:54.124771740 (18h11m48.555851871s) [noise=true , address=77.90.31.18:4806 , piece_id=QCURQYEJL22LBT2YFBR62RIHH7VNSGL7GERP66GKHXK3JP5WLVZQ, seg_pos={0 0}, upload=18998]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: write tcp 188.166.225.170:55536->77.90.31.18:4806: use of closed network connection; write tcp 188.166.225.170:55536->77.90.31.18:4806: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55536->77.90.31.18:4806: use of closed network connection Aug 9 11:37:54.124817217 (18h11m48.555897348s) [noise=true , address=77.90.31.15:4697 , piece_id=Q26SRGZO2YBJWPDIG4U422EICE6XEXPLHFHXLKFUBKMMJAZ4W7UQ, seg_pos={0 0}, upload=18998]: Done uploading piece. err: ecclient: upload failed (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, address:77.90.31.15:4697): protocol: write tcp 188.166.225.170:49618->77.90.31.15:4697: use of closed network connection; write tcp 188.166.225.170:49618->77.90.31.15:4697: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49618->77.90.31.15:4697: use of closed network connection Aug 9 11:37:54.124833387 (18h11m48.555913559s) [noise=true , address=45.91.116.6:4973 , piece_id=MHDNXTVKFMLRKGS7JIPL4U5VQMBXGP4EYM7JEGPSK4U3GIJ5JEHQ, seg_pos={0 0}, upload=18998]: Done uploading piece. err: ecclient: upload failed (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, address:45.91.116.6:4973): protocol: write tcp 188.166.225.170:59438->45.91.116.6:4973: use of closed network connection; write tcp 188.166.225.170:59438->45.91.116.6:4973: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59438->45.91.116.6:4973: use of closed network connection Aug 9 11:37:54.124840752 (18h11m48.555920883s) [noise=true , address=77.90.31.17:4772 , piece_id=MGZPHOODRIOTGABQJHNIP5FGF4XN6A723ZHUBKI6KNAJDKFWWOJA, seg_pos={0 0}, upload=18998]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: write tcp 188.166.225.170:42704->77.90.31.17:4772: use of closed network connection; write tcp 188.166.225.170:42704->77.90.31.17:4772: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42704->77.90.31.17:4772: use of closed network connection Aug 9 11:37:54.124845251 (18h11m48.555925382s) [noise=true , address=45.91.116.8:4047 , piece_id=B27B2QKZXICP4DHTDTCYSZZ7RN2PPCMKSRQBKRK5JJ4BE4RNWQAQ, seg_pos={0 0}, upload=18998]: Done uploading piece. err: ecclient: upload failed (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN, address:45.91.116.8:4047): protocol: write tcp 188.166.225.170:40794->45.91.116.8:4047: use of closed network connection; write tcp 188.166.225.170:40794->45.91.116.8:4047: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40794->45.91.116.8:4047: use of closed network connection Aug 9 11:37:54.124866321 (18h11m48.555946492s) [seg_pos={0 0}, upload=18998]: Done waiting for segment. successful: 6 optimal: 13 errs: <nil> Aug 9 11:37:54.353990811 (18h11m48.785070942s) [upload=18998]: Done uploading segments... Aug 9 11:37:54.354028673 (18h11m48.785108804s) [upload=18998]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 6) Duration: 5.650403818s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 3 out of 60 is failed (threshold=3) [....................................X............X.X........] duration: 5.814059621s FAIL (2025-08-09 12:05:27 UTC) orbiter warning: operation: ListObjects ======ERROR====== unable to find key in bucket: sj://uptime-checks-placement-20/orbiter_us_0/4M ======LOGS====== Aug 9 12:00:23.108483377 (18h34m17.539563508s) [upload=19402]: Starting upload Aug 9 12:00:23.108636078 (18h34m17.53971621s) [upload=19402]: Done starting upload Aug 9 12:00:23.108746840 (18h34m17.539826981s) [upload=19402]: Uploading segments... Aug 9 12:00:23.108904442 (18h34m17.539984573s) [upload=19402]: Got next segment. Inline: false Aug 9 12:00:23.108970418 (18h34m17.540050599s) [upload=19402]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 9 12:00:23.108977743 (18h34m17.540057904s) [upload=19402]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 9 12:00:23.579075368 (18h34m18.010155499s) [seg_pos={0 0}, upload=19402]: Begin upload segment... Aug 9 12:00:23.579127947 (18h34m18.010208078s) [seg_pos={0 0}, upload=19402]: Done begin upload segment. Aug 9 12:00:23.579157074 (18h34m18.010237205s) [noise=true , address=104.36.82.44:4899 , piece_id=KIQ5AFZ2FIUJ3XIG5ZQBJYITK5CD4ZDW5SYOUPFIYAOZBPENWFFA, seg_pos={0 0}, upload=19402]: Uploading piece... Aug 9 12:00:23.579206939 (18h34m18.01028707s) [noise=true , address=104.36.82.55:4199 , piece_id=6D4WT6KDBSMMCPXONV74YG4IXFEXPXU2FQTCL2W2NPZ2ERVMBUMQ, seg_pos={0 0}, upload=19402]: Uploading piece... Aug 9 12:00:23.579552951 (18h34m18.010633082s) [noise=true , address=64.49.13.139:4469 , piece_id=UTZT7WZ3LQ6LOZ3C6IXV5M7L3APATJOYDRZAMWUEYCZP4I34THCA, seg_pos={0 0}, upload=19402]: Uploading piece... Aug 9 12:00:23.579921666 (18h34m18.011001797s) [noise=true , address=104.36.82.59:4327 , piece_id=WJJFOV44IRLH2E6BFC46ALBJXD54ML4HQOFSUDFNJ3OJUXASSTZQ, seg_pos={0 0}, upload=19402]: Uploading piece... Aug 9 12:00:23.580275103 (18h34m18.011355234s) [noise=true , address=104.36.82.13:4197 , piece_id=NEMRRYQPHXM73GSZVEETU7UNMUDQVNKKSLKQTRSBHBKPQ35KLLPQ, seg_pos={0 0}, upload=19402]: Uploading piece... Aug 9 12:00:23.580699144 (18h34m18.011779275s) [noise=true , address=104.36.82.23:4454 , piece_id=VNKC653W6NNZTTH6CRDK2LSD5XFLLGJGYOOIELABDCKC364DITDA, seg_pos={0 0}, upload=19402]: Uploading piece... Aug 9 12:00:23.580803934 (18h34m18.011884065s) [noise=true , address=104.36.82.61:4396 , piece_id=LJ73GMLI4ZDNLDSGIRT4GRBU56YFUOBTOC7OW3NS4TRWSUXMQCHA, seg_pos={0 0}, upload=19402]: Uploading piece... Aug 9 12:00:23.581071286 (18h34m18.012151417s) [noise=true , address=104.36.82.45:4934 , piece_id=K7XWI37JOPKYII6BFSO6AZT73S3FYOH5JKCJXOM5O5SCWAG62OIQ, seg_pos={0 0}, upload=19402]: Uploading piece... Aug 9 12:00:23.581305595 (18h34m18.012385726s) [noise=true , address=104.36.82.60:4354 , piece_id=HR52C2IWZNCL2XI553EUKTS37R5N4ILVW34GJSPWJV3Z3W7KKGWA, seg_pos={0 0}, upload=19402]: Uploading piece... Aug 9 12:00:23.581541305 (18h34m18.012621436s) [noise=true , address=45.91.116.12:4243 , piece_id=FZW53SVI3PFJ3M6WQE74FPB737JEBGBGA2YB53XCU5DEPEUQHU7Q, seg_pos={0 0}, upload=19402]: Uploading piece... Aug 9 12:00:23.581620086 (18h34m18.012700217s) [noise=true , address=104.36.82.58:4268 , piece_id=QLMKG6PZ4XYJJUTOHFW7EWTFGTST5HYLTHP5MMX6X3AJJJAZSZ3Q, seg_pos={0 0}, upload=19402]: Uploading piece... Aug 9 12:00:23.581922384 (18h34m18.013002515s) [noise=true , address=104.36.82.9:4031 , piece_id=LLSOV26QMVJRDNGWSOKPT4MIGAMTHU6AIADGGJNG6526XAR3SF6Q, seg_pos={0 0}, upload=19402]: Uploading piece... Aug 9 12:00:23.582143177 (18h34m18.013223308s) [noise=true , address=104.36.82.60:4358 , piece_id=WPW7FORNXWY6DOAXVH775D4W3VFOH7PRMQO5UXKJNIFFCRVSE43A, seg_pos={0 0}, upload=19402]: Uploading piece... Aug 9 12:00:23.582347538 (18h34m18.013427669s) [noise=true , address=104.36.82.14:4242 , piece_id=2JREAT5B4ZOSHBJPMPKVQOW2VDMLQIDURO7KTYRPFJAM6U5K57WQ, seg_pos={0 0}, upload=19402]: Uploading piece... Aug 9 12:00:23.582596004 (18h34m18.013676135s) [noise=true , address=64.49.13.170:4726 , piece_id=A2OWN4ZBVO3MZKLRAHERFMYAZDSLB4HMYESALCEVEH66LGUZEQOQ, seg_pos={0 0}, upload=19402]: Uploading piece... Aug 9 12:00:23.582774395 (18h34m18.013854526s) [noise=true , address=104.36.82.46:4484 , piece_id=FQHBXDSLXK2GIOSY4V5QBBPLOH7ZD3KHJVXR5IFYPMICQCWH7UFA, seg_pos={0 0}, upload=19402]: Uploading piece... Aug 9 12:00:23.583063899 (18h34m18.01414403s) [noise=true , address=77.90.31.12:4604 , piece_id=HPMFA765VZOJGNXPWE5SCA4AAK7QB4HFGUSYFQWOFLVZTU6PSUMA, seg_pos={0 0}, upload=19402]: Uploading piece... Aug 9 12:00:23.583138481 (18h34m18.014218612s) [noise=true , address=104.36.82.5:4845 , piece_id=5AJNZJD6MFKE3U6EZ27IEXQEO4KOUGSXA7EBCDSWVTVFYD4AJTHQ, seg_pos={0 0}, upload=19402]: Uploading piece... Aug 9 12:00:24.683477824 (18h34m19.114557955s) [upload=19402]: Next returned nil segment Aug 9 12:00:24.683489506 (18h34m19.114569637s) [upload=19402]: Waiting for error group managing segments... Aug 9 12:00:24.955322603 (18h34m19.386402734s) [noise=true , address=64.49.13.170:4726 , piece_id=A2OWN4ZBVO3MZKLRAHERFMYAZDSLB4HMYESALCEVEH66LGUZEQOQ, seg_pos={0 0}, upload=19402]: Done uploading piece. err: <nil> Aug 9 12:00:24.969832404 (18h34m19.400912535s) [noise=true , address=64.49.13.139:4469 , piece_id=UTZT7WZ3LQ6LOZ3C6IXV5M7L3APATJOYDRZAMWUEYCZP4I34THCA, seg_pos={0 0}, upload=19402]: Done uploading piece. err: <nil> Aug 9 12:00:25.516280300 (18h34m19.947360431s) [noise=true , address=104.36.82.55:4199 , piece_id=6D4WT6KDBSMMCPXONV74YG4IXFEXPXU2FQTCL2W2NPZ2ERVMBUMQ, seg_pos={0 0}, upload=19402]: Done uploading piece. err: <nil> Aug 9 12:00:25.525386162 (18h34m19.956466293s) [noise=true , address=104.36.82.46:4484 , piece_id=FQHBXDSLXK2GIOSY4V5QBBPLOH7ZD3KHJVXR5IFYPMICQCWH7UFA, seg_pos={0 0}, upload=19402]: Done uploading piece. err: <nil> Aug 9 12:00:25.526709204 (18h34m19.957789335s) [noise=true , address=104.36.82.60:4358 , piece_id=WPW7FORNXWY6DOAXVH775D4W3VFOH7PRMQO5UXKJNIFFCRVSE43A, seg_pos={0 0}, upload=19402]: Done uploading piece. err: <nil> Aug 9 12:00:25.530790893 (18h34m19.961871024s) [noise=true , address=104.36.82.44:4899 , piece_id=KIQ5AFZ2FIUJ3XIG5ZQBJYITK5CD4ZDW5SYOUPFIYAOZBPENWFFA, seg_pos={0 0}, upload=19402]: Done uploading piece. err: <nil> Aug 9 12:00:25.531900687 (18h34m19.962980818s) [noise=true , address=104.36.82.58:4268 , piece_id=QLMKG6PZ4XYJJUTOHFW7EWTFGTST5HYLTHP5MMX6X3AJJJAZSZ3Q, seg_pos={0 0}, upload=19402]: Done uploading piece. err: <nil> Aug 9 12:00:25.536797967 (18h34m19.967878098s) [noise=true , address=104.36.82.9:4031 , piece_id=LLSOV26QMVJRDNGWSOKPT4MIGAMTHU6AIADGGJNG6526XAR3SF6Q, seg_pos={0 0}, upload=19402]: Done uploading piece. err: <nil> Aug 9 12:00:25.545685641 (18h34m19.976765772s) [noise=true , address=104.36.82.60:4354 , piece_id=HR52C2IWZNCL2XI553EUKTS37R5N4ILVW34GJSPWJV3Z3W7KKGWA, seg_pos={0 0}, upload=19402]: Done uploading piece. err: <nil> Aug 9 12:00:25.555951362 (18h34m19.987031493s) [noise=true , address=104.36.82.61:4396 , piece_id=LJ73GMLI4ZDNLDSGIRT4GRBU56YFUOBTOC7OW3NS4TRWSUXMQCHA, seg_pos={0 0}, upload=19402]: Done uploading piece. err: <nil> Aug 9 12:00:25.566816712 (18h34m19.997896843s) [noise=true , address=104.36.82.5:4845 , piece_id=5AJNZJD6MFKE3U6EZ27IEXQEO4KOUGSXA7EBCDSWVTVFYD4AJTHQ, seg_pos={0 0}, upload=19402]: Done uploading piece. err: <nil> Aug 9 12:00:25.585861129 (18h34m20.01694126s) [noise=true , address=104.36.82.45:4934 , piece_id=K7XWI37JOPKYII6BFSO6AZT73S3FYOH5JKCJXOM5O5SCWAG62OIQ, seg_pos={0 0}, upload=19402]: Done uploading piece. err: <nil> Aug 9 12:00:25.594196727 (18h34m20.025276858s) [noise=true , address=45.91.116.12:4243 , piece_id=FZW53SVI3PFJ3M6WQE74FPB737JEBGBGA2YB53XCU5DEPEUQHU7Q, seg_pos={0 0}, upload=19402]: Done uploading piece. err: <nil> Aug 9 12:00:25.606123457 (18h34m20.037203588s) [noise=true , address=77.90.31.12:4604 , piece_id=HPMFA765VZOJGNXPWE5SCA4AAK7QB4HFGUSYFQWOFLVZTU6PSUMA, seg_pos={0 0}, upload=19402]: Done uploading piece. err: <nil> Aug 9 12:00:25.606140419 (18h34m20.03722055s) [seg_pos={0 0}, upload=19402]: Segment reached optimal threshold of 14 pieces. Aug 9 12:00:25.606241462 (18h34m20.037321593s) [noise=true , address=104.36.82.13:4197 , piece_id=NEMRRYQPHXM73GSZVEETU7UNMUDQVNKKSLKQTRSBHBKPQ35KLLPQ, seg_pos={0 0}, upload=19402]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1cB89C5xnya9WGD9dGBsSouqkZH3nLqytrg9Qo4UH2kdFonTjM): protocol: expected piece hash; context canceled; EOF Aug 9 12:00:25.606346553 (18h34m20.037426675s) [noise=true , address=104.36.82.14:4242 , piece_id=2JREAT5B4ZOSHBJPMPKVQOW2VDMLQIDURO7KTYRPFJAM6U5K57WQ, seg_pos={0 0}, upload=19402]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1inR8rjUp4fRQWcaD2S7MRxveHyFxKoutWoEiuKQG3C895PaMz): protocol: expected piece hash; context canceled; EOF Aug 9 12:00:25.606405596 (18h34m20.037485727s) [noise=true , address=104.36.82.23:4454 , piece_id=VNKC653W6NNZTTH6CRDK2LSD5XFLLGJGYOOIELABDCKC364DITDA, seg_pos={0 0}, upload=19402]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1UCkuYGdVYLoR8oc1jpCZCN2smNfVqFwC8a7q95fJb8haB3MY2): protocol: expected piece hash; context canceled; EOF Aug 9 12:00:25.606463838 (18h34m20.037543969s) [noise=true , address=104.36.82.59:4327 , piece_id=WJJFOV44IRLH2E6BFC46ALBJXD54ML4HQOFSUDFNJ3OJUXASSTZQ, seg_pos={0 0}, upload=19402]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1V4C6AWGbyYQWfinoSj7iKovHeTJ9hqypyC5U9rZ8pveXBBfCY): protocol: expected piece hash; context canceled; EOF Aug 9 12:00:25.606478186 (18h34m20.037558317s) [seg_pos={0 0}, upload=19402]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 9 12:00:25.606535966 (18h34m20.037616097s) [upload=19402]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 9 12:00:25.606540855 (18h34m20.037620976s) [upload=19402]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 9 12:00:25.913625167 (18h34m20.344705298s) [upload=19402]: Done uploading segments... Aug 9 12:00:25.913644565 (18h34m20.344724696s) [upload=19402]: Upload finished. err: <nil> Duration: 5.814085821s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................X.....................................] duration: 5.819143306s FAIL (2025-08-09 12:05:06 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 6) ======LOGS====== ADHZNIB6LBP5VQHFEGSYM3YNNZWNM2N4UU7JTCQ, seg_pos={0 0}, upload=18998]: Uploading piece... Aug 9 11:37:25.005421531 (18h11m19.436501662s) [noise=true , address=45.91.116.4:4940 , piece_id=M33ZGUCEF23HDGQZE5F5KLWQ43NHC7E7LKOVVGQ6P6TM2HFGB4WQ, seg_pos={0 0}, upload=18998]: Uploading piece... Aug 9 11:37:25.005614329 (18h11m19.43669446s) [noise=true , address=45.91.116.8:4047 , piece_id=B27B2QKZXICP4DHTDTCYSZZ7RN2PPCMKSRQBKRK5JJ4BE4RNWQAQ, seg_pos={0 0}, upload=18998]: Uploading piece... Aug 9 11:37:25.005727536 (18h11m19.436807667s) [noise=true , address=45.91.116.12:4225 , piece_id=VFK2SMVTBT76D7YLQWGRA6TF47JD3B7J7F5NFR4E2MQVNGNHYTAQ, seg_pos={0 0}, upload=18998]: Uploading piece... Aug 9 11:37:25.006281517 (18h11m19.437361648s) [noise=true , address=104.36.82.7:4909 , piece_id=OGG5ZCXSLIM5YKMM7K6KLKLWGOTX376DUTDT3567OB3AJIBPBKDQ, seg_pos={0 0}, upload=18998]: Uploading piece... Aug 9 11:37:25.006632558 (18h11m19.437712679s) [noise=true , address=45.91.116.13:4155 , piece_id=K4CGD4OCR2B6ZWAZHB5MDBECVJB3AC2HDXS2TUVAVGIWQYJEARLA, seg_pos={0 0}, upload=18998]: Uploading piece... Aug 9 11:37:25.006996264 (18h11m19.438076395s) [noise=true , address=77.90.31.18:4806 , piece_id=QCURQYEJL22LBT2YFBR62RIHH7VNSGL7GERP66GKHXK3JP5WLVZQ, seg_pos={0 0}, upload=18998]: Uploading piece... Aug 9 11:37:25.007245231 (18h11m19.438325362s) [noise=true , address=104.36.82.3:4765 , piece_id=HBLTTPZYK5ZXNG2XV53QVJA3NPX3YWKEAAAFVMDXUO6BLR3QKCWQ, seg_pos={0 0}, upload=18998]: Uploading piece... Aug 9 11:37:25.007475481 (18h11m19.438555612s) [noise=true , address=45.91.116.6:4973 , piece_id=MHDNXTVKFMLRKGS7JIPL4U5VQMBXGP4EYM7JEGPSK4U3GIJ5JEHQ, seg_pos={0 0}, upload=18998]: Uploading piece... Aug 9 11:37:25.007873984 (18h11m19.438954115s) [noise=true , address=45.91.116.7:4010 , piece_id=5G3O552GD5XO7SNIHST623QJBWJF2C7B6EN7ISGHHY3FYV6QL2DQ, seg_pos={0 0}, upload=18998]: Uploading piece... Aug 9 11:37:26.733124951 (18h11m21.164205082s) [upload=18998]: Next returned nil segment Aug 9 11:37:26.733142194 (18h11m21.164222325s) [upload=18998]: Waiting for error group managing segments... Aug 9 11:37:27.100757697 (18h11m21.531837848s) [noise=true , address=104.36.82.14:4222 , piece_id=Z4X6RJ5HBAYGUNKT6DXWSS2TEG2OUFPMH3NWYGA7D26XIFH46UAA, seg_pos={0 0}, upload=18998]: Done uploading piece. err: <nil> Aug 9 11:37:27.106781913 (18h11m21.537862064s) [noise=true , address=104.36.82.7:4912 , piece_id=MRA5X6N6MDQ2TADHZNIB6LBP5VQHFEGSYM3YNNZWNM2N4UU7JTCQ, seg_pos={0 0}, upload=18998]: Done uploading piece. err: <nil> Aug 9 11:37:27.136679105 (18h11m21.567759236s) [noise=true , address=104.36.82.5:4838 , piece_id=T3WCWCLB4WAXQY3S6VCFHLKYK6IY2Q6P2776CLGYS4QOFTXMYVEQ, seg_pos={0 0}, upload=18998]: Done uploading piece. err: <nil> Aug 9 11:37:27.139126447 (18h11m21.570206578s) [noise=true , address=104.36.82.3:4765 , piece_id=HBLTTPZYK5ZXNG2XV53QVJA3NPX3YWKEAAAFVMDXUO6BLR3QKCWQ, seg_pos={0 0}, upload=18998]: Done uploading piece. err: <nil> Aug 9 11:37:27.183514474 (18h11m21.614594606s) [noise=true , address=104.36.82.7:4909 , piece_id=OGG5ZCXSLIM5YKMM7K6KLKLWGOTX376DUTDT3567OB3AJIBPBKDQ, seg_pos={0 0}, upload=18998]: Done uploading piece. err: <nil> Aug 9 11:37:53.489259032 (18h11m47.920339163s) [noise=true , address=45.91.116.13:4155 , piece_id=K4CGD4OCR2B6ZWAZHB5MDBECVJB3AC2HDXS2TUVAVGIWQYJEARLA, seg_pos={0 0}, upload=18998]: Done uploading piece. err: <nil> Aug 9 11:37:54.124525138 (18h11m48.555605259s) [noise=true , address=45.91.116.7:4010 , piece_id=5G3O552GD5XO7SNIHST623QJBWJF2C7B6EN7ISGHHY3FYV6QL2DQ, seg_pos={0 0}, upload=18998]: Done uploading piece. err: ecclient: upload failed (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w, address:45.91.116.7:4010): protocol: write tcp 188.166.225.170:54960->45.91.116.7:4010: use of closed network connection; write tcp 188.166.225.170:54960->45.91.116.7:4010: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54960->45.91.116.7:4010: use of closed network connection Aug 9 11:37:54.124581035 (18h11m48.555661166s) [noise=true , address=77.90.31.10:4520 , piece_id=YRBBWM54FGU3B74JQ7EOAO7VSGVFXAVT4SNZ6KN3IQFIHIEVV5HA, seg_pos={0 0}, upload=18998]: Done uploading piece. err: ecclient: upload failed (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp, address:77.90.31.10:4520): protocol: write tcp 188.166.225.170:39560->77.90.31.10:4520: use of closed network connection; write tcp 188.166.225.170:39560->77.90.31.10:4520: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39560->77.90.31.10:4520: use of closed network connection Aug 9 11:37:54.124615480 (18h11m48.555695652s) [noise=true , address=45.91.116.12:4225 , piece_id=VFK2SMVTBT76D7YLQWGRA6TF47JD3B7J7F5NFR4E2MQVNGNHYTAQ, seg_pos={0 0}, upload=18998]: Done uploading piece. err: ecclient: upload failed (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy, address:45.91.116.12:4225): protocol: write tcp 188.166.225.170:38054->45.91.116.12:4225: use of closed network connection; write tcp 188.166.225.170:38054->45.91.116.12:4225: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38054->45.91.116.12:4225: use of closed network connection Aug 9 11:37:54.124643875 (18h11m48.555724016s) [noise=true , address=45.91.116.4:4940 , piece_id=M33ZGUCEF23HDGQZE5F5KLWQ43NHC7E7LKOVVGQ6P6TM2HFGB4WQ, seg_pos={0 0}, upload=18998]: Done uploading piece. err: ecclient: upload failed (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe, address:45.91.116.4:4940): protocol: write tcp 188.166.225.170:40630->45.91.116.4:4940: use of closed network connection; write tcp 188.166.225.170:40630->45.91.116.4:4940: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40630->45.91.116.4:4940: use of closed network connection Aug 9 11:37:54.124693159 (18h11m48.55577335s) [noise=true , address=45.91.116.14:4189 , piece_id=XVQMTYHB6H3IKSERYT2BEB4NP42KV6FRE6J2MKKU5WEOCVBHYIWA, seg_pos={0 0}, upload=18998]: Done uploading piece. err: ecclient: upload failed (node:1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N, address:45.91.116.14:4189): protocol: write tcp 188.166.225.170:37650->45.91.116.14:4189: use of closed network connection; write tcp 188.166.225.170:37650->45.91.116.14:4189: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37650->45.91.116.14:4189: use of closed network connection Aug 9 11:37:54.124771740 (18h11m48.555851871s) [noise=true , address=77.90.31.18:4806 , piece_id=QCURQYEJL22LBT2YFBR62RIHH7VNSGL7GERP66GKHXK3JP5WLVZQ, seg_pos={0 0}, upload=18998]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: write tcp 188.166.225.170:55536->77.90.31.18:4806: use of closed network connection; write tcp 188.166.225.170:55536->77.90.31.18:4806: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55536->77.90.31.18:4806: use of closed network connection Aug 9 11:37:54.124817217 (18h11m48.555897348s) [noise=true , address=77.90.31.15:4697 , piece_id=Q26SRGZO2YBJWPDIG4U422EICE6XEXPLHFHXLKFUBKMMJAZ4W7UQ, seg_pos={0 0}, upload=18998]: Done uploading piece. err: ecclient: upload failed (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, address:77.90.31.15:4697): protocol: write tcp 188.166.225.170:49618->77.90.31.15:4697: use of closed network connection; write tcp 188.166.225.170:49618->77.90.31.15:4697: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49618->77.90.31.15:4697: use of closed network connection Aug 9 11:37:54.124833387 (18h11m48.555913559s) [noise=true , address=45.91.116.6:4973 , piece_id=MHDNXTVKFMLRKGS7JIPL4U5VQMBXGP4EYM7JEGPSK4U3GIJ5JEHQ, seg_pos={0 0}, upload=18998]: Done uploading piece. err: ecclient: upload failed (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, address:45.91.116.6:4973): protocol: write tcp 188.166.225.170:59438->45.91.116.6:4973: use of closed network connection; write tcp 188.166.225.170:59438->45.91.116.6:4973: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59438->45.91.116.6:4973: use of closed network connection Aug 9 11:37:54.124840752 (18h11m48.555920883s) [noise=true , address=77.90.31.17:4772 , piece_id=MGZPHOODRIOTGABQJHNIP5FGF4XN6A723ZHUBKI6KNAJDKFWWOJA, seg_pos={0 0}, upload=18998]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: write tcp 188.166.225.170:42704->77.90.31.17:4772: use of closed network connection; write tcp 188.166.225.170:42704->77.90.31.17:4772: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42704->77.90.31.17:4772: use of closed network connection Aug 9 11:37:54.124845251 (18h11m48.555925382s) [noise=true , address=45.91.116.8:4047 , piece_id=B27B2QKZXICP4DHTDTCYSZZ7RN2PPCMKSRQBKRK5JJ4BE4RNWQAQ, seg_pos={0 0}, upload=18998]: Done uploading piece. err: ecclient: upload failed (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN, address:45.91.116.8:4047): protocol: write tcp 188.166.225.170:40794->45.91.116.8:4047: use of closed network connection; write tcp 188.166.225.170:40794->45.91.116.8:4047: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40794->45.91.116.8:4047: use of closed network connection Aug 9 11:37:54.124866321 (18h11m48.555946492s) [seg_pos={0 0}, upload=18998]: Done waiting for segment. successful: 6 optimal: 13 errs: <nil> Aug 9 11:37:54.353990811 (18h11m48.785070942s) [upload=18998]: Done uploading segments... Aug 9 11:37:54.354028673 (18h11m48.785108804s) [upload=18998]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 6) Duration: 5.81917092s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 3 out of 60 is failed (threshold=3) [.......................X..................X.........X.......] duration: 1.882833655s FAIL (2025-08-09 12:05:05 UTC) orbiter warning: operation: ListObjects ======ERROR====== unable to find key in bucket: sj://uptime-checks-placement-20/orbiter_us_0/1K ======LOGS====== Aug 9 11:59:31.819436107 (18h33m26.250516238s) [upload=19388]: Starting upload Aug 9 11:59:31.819492355 (18h33m26.250572516s) [upload=19388]: Done starting upload Aug 9 11:59:31.819563080 (18h33m26.250643211s) [upload=19388]: Uploading segments... Aug 9 11:59:31.819616853 (18h33m26.250696984s) [upload=19388]: Got next segment. Inline: true Aug 9 11:59:31.819621552 (18h33m26.250701683s) [upload=19388]: Waiting for error group managing segments... Aug 9 11:59:31.819660466 (18h33m26.250740597s) [upload=19388]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 9 11:59:31.819665206 (18h33m26.250745327s) [upload=19388]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 9 11:59:31.819667851 (18h33m26.250747972s) [upload=19388]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 9 11:59:32.242597809 (18h33m26.67367794s) [upload=19388]: Done uploading segments... Aug 9 11:59:32.242615142 (18h33m26.673695263s) [upload=19388]: Upload finished. err: <nil> Duration:* 1.882862511s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 3 out of 60 is failed (threshold=3) [.....................................X............X.X.......] duration: 5.869548088s FAIL (2025-08-09 12:04:45 UTC) orbiter warning: operation: ListObjects ======ERROR====== unable to find key in bucket: sj://uptime-checks-placement-20/orbiter_us_0/4M ======LOGS====== Aug 9 12:00:23.108483377 (18h34m17.539563508s) [upload=19402]: Starting upload Aug 9 12:00:23.108636078 (18h34m17.53971621s) [upload=19402]: Done starting upload Aug 9 12:00:23.108746840 (18h34m17.539826981s) [upload=19402]: Uploading segments... Aug 9 12:00:23.108904442 (18h34m17.539984573s) [upload=19402]: Got next segment. Inline: false Aug 9 12:00:23.108970418 (18h34m17.540050599s) [upload=19402]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 9 12:00:23.108977743 (18h34m17.540057904s) [upload=19402]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 9 12:00:23.579075368 (18h34m18.010155499s) [seg_pos={0 0}, upload=19402]: Begin upload segment... Aug 9 12:00:23.579127947 (18h34m18.010208078s) [seg_pos={0 0}, upload=19402]: Done begin upload segment. Aug 9 12:00:23.579157074 (18h34m18.010237205s) [noise=true , address=104.36.82.44:4899 , piece_id=KIQ5AFZ2FIUJ3XIG5ZQBJYITK5CD4ZDW5SYOUPFIYAOZBPENWFFA, seg_pos={0 0}, upload=19402]: Uploading piece... Aug 9 12:00:23.579206939 (18h34m18.01028707s) [noise=true , address=104.36.82.55:4199 , piece_id=6D4WT6KDBSMMCPXONV74YG4IXFEXPXU2FQTCL2W2NPZ2ERVMBUMQ, seg_pos={0 0}, upload=19402]: Uploading piece... Aug 9 12:00:23.579552951 (18h34m18.010633082s) [noise=true , address=64.49.13.139:4469 , piece_id=UTZT7WZ3LQ6LOZ3C6IXV5M7L3APATJOYDRZAMWUEYCZP4I34THCA, seg_pos={0 0}, upload=19402]: Uploading piece... Aug 9 12:00:23.579921666 (18h34m18.011001797s) [noise=true , address=104.36.82.59:4327 , piece_id=WJJFOV44IRLH2E6BFC46ALBJXD54ML4HQOFSUDFNJ3OJUXASSTZQ, seg_pos={0 0}, upload=19402]: Uploading piece... Aug 9 12:00:23.580275103 (18h34m18.011355234s) [noise=true , address=104.36.82.13:4197 , piece_id=NEMRRYQPHXM73GSZVEETU7UNMUDQVNKKSLKQTRSBHBKPQ35KLLPQ, seg_pos={0 0}, upload=19402]: Uploading piece... Aug 9 12:00:23.580699144 (18h34m18.011779275s) [noise=true , address=104.36.82.23:4454 , piece_id=VNKC653W6NNZTTH6CRDK2LSD5XFLLGJGYOOIELABDCKC364DITDA, seg_pos={0 0}, upload=19402]: Uploading piece... Aug 9 12:00:23.580803934 (18h34m18.011884065s) [noise=true , address=104.36.82.61:4396 , piece_id=LJ73GMLI4ZDNLDSGIRT4GRBU56YFUOBTOC7OW3NS4TRWSUXMQCHA, seg_pos={0 0}, upload=19402]: Uploading piece... Aug 9 12:00:23.581071286 (18h34m18.012151417s) [noise=true , address=104.36.82.45:4934 , piece_id=K7XWI37JOPKYII6BFSO6AZT73S3FYOH5JKCJXOM5O5SCWAG62OIQ, seg_pos={0 0}, upload=19402]: Uploading piece... Aug 9 12:00:23.581305595 (18h34m18.012385726s) [noise=true , address=104.36.82.60:4354 , piece_id=HR52C2IWZNCL2XI553EUKTS37R5N4ILVW34GJSPWJV3Z3W7KKGWA, seg_pos={0 0}, upload=19402]: Uploading piece... Aug 9 12:00:23.581541305 (18h34m18.012621436s) [noise=true , address=45.91.116.12:4243 , piece_id=FZW53SVI3PFJ3M6WQE74FPB737JEBGBGA2YB53XCU5DEPEUQHU7Q, seg_pos={0 0}, upload=19402]: Uploading piece... Aug 9 12:00:23.581620086 (18h34m18.012700217s) [noise=true , address=104.36.82.58:4268 , piece_id=QLMKG6PZ4XYJJUTOHFW7EWTFGTST5HYLTHP5MMX6X3AJJJAZSZ3Q, seg_pos={0 0}, upload=19402]: Uploading piece... Aug 9 12:00:23.581922384 (18h34m18.013002515s) [noise=true , address=104.36.82.9:4031 , piece_id=LLSOV26QMVJRDNGWSOKPT4MIGAMTHU6AIADGGJNG6526XAR3SF6Q, seg_pos={0 0}, upload=19402]: Uploading piece... Aug 9 12:00:23.582143177 (18h34m18.013223308s) [noise=true , address=104.36.82.60:4358 , piece_id=WPW7FORNXWY6DOAXVH775D4W3VFOH7PRMQO5UXKJNIFFCRVSE43A, seg_pos={0 0}, upload=19402]: Uploading piece... Aug 9 12:00:23.582347538 (18h34m18.013427669s) [noise=true , address=104.36.82.14:4242 , piece_id=2JREAT5B4ZOSHBJPMPKVQOW2VDMLQIDURO7KTYRPFJAM6U5K57WQ, seg_pos={0 0}, upload=19402]: Uploading piece... Aug 9 12:00:23.582596004 (18h34m18.013676135s) [noise=true , address=64.49.13.170:4726 , piece_id=A2OWN4ZBVO3MZKLRAHERFMYAZDSLB4HMYESALCEVEH66LGUZEQOQ, seg_pos={0 0}, upload=19402]: Uploading piece... Aug 9 12:00:23.582774395 (18h34m18.013854526s) [noise=true , address=104.36.82.46:4484 , piece_id=FQHBXDSLXK2GIOSY4V5QBBPLOH7ZD3KHJVXR5IFYPMICQCWH7UFA, seg_pos={0 0}, upload=19402]: Uploading piece... Aug 9 12:00:23.583063899 (18h34m18.01414403s) [noise=true , address=77.90.31.12:4604 , piece_id=HPMFA765VZOJGNXPWE5SCA4AAK7QB4HFGUSYFQWOFLVZTU6PSUMA, seg_pos={0 0}, upload=19402]: Uploading piece... Aug 9 12:00:23.583138481 (18h34m18.014218612s) [noise=true , address=104.36.82.5:4845 , piece_id=5AJNZJD6MFKE3U6EZ27IEXQEO4KOUGSXA7EBCDSWVTVFYD4AJTHQ, seg_pos={0 0}, upload=19402]: Uploading piece... Aug 9 12:00:24.683477824 (18h34m19.114557955s) [upload=19402]: Next returned nil segment Aug 9 12:00:24.683489506 (18h34m19.114569637s) [upload=19402]: Waiting for error group managing segments... Aug 9 12:00:24.955322603 (18h34m19.386402734s) [noise=true , address=64.49.13.170:4726 , piece_id=A2OWN4ZBVO3MZKLRAHERFMYAZDSLB4HMYESALCEVEH66LGUZEQOQ, seg_pos={0 0}, upload=19402]: Done uploading piece. err: <nil> Aug 9 12:00:24.969832404 (18h34m19.400912535s) [noise=true , address=64.49.13.139:4469 , piece_id=UTZT7WZ3LQ6LOZ3C6IXV5M7L3APATJOYDRZAMWUEYCZP4I34THCA, seg_pos={0 0}, upload=19402]: Done uploading piece. err: <nil> Aug 9 12:00:25.516280300 (18h34m19.947360431s) [noise=true , address=104.36.82.55:4199 , piece_id=6D4WT6KDBSMMCPXONV74YG4IXFEXPXU2FQTCL2W2NPZ2ERVMBUMQ, seg_pos={0 0}, upload=19402]: Done uploading piece. err: <nil> Aug 9 12:00:25.525386162 (18h34m19.956466293s) [noise=true , address=104.36.82.46:4484 , piece_id=FQHBXDSLXK2GIOSY4V5QBBPLOH7ZD3KHJVXR5IFYPMICQCWH7UFA, seg_pos={0 0}, upload=19402]: Done uploading piece. err: <nil> Aug 9 12:00:25.526709204 (18h34m19.957789335s) [noise=true , address=104.36.82.60:4358 , piece_id=WPW7FORNXWY6DOAXVH775D4W3VFOH7PRMQO5UXKJNIFFCRVSE43A, seg_pos={0 0}, upload=19402]: Done uploading piece. err: <nil> Aug 9 12:00:25.530790893 (18h34m19.961871024s) [noise=true , address=104.36.82.44:4899 , piece_id=KIQ5AFZ2FIUJ3XIG5ZQBJYITK5CD4ZDW5SYOUPFIYAOZBPENWFFA, seg_pos={0 0}, upload=19402]: Done uploading piece. err: <nil> Aug 9 12:00:25.531900687 (18h34m19.962980818s) [noise=true , address=104.36.82.58:4268 , piece_id=QLMKG6PZ4XYJJUTOHFW7EWTFGTST5HYLTHP5MMX6X3AJJJAZSZ3Q, seg_pos={0 0}, upload=19402]: Done uploading piece. err: <nil> Aug 9 12:00:25.536797967 (18h34m19.967878098s) [noise=true , address=104.36.82.9:4031 , piece_id=LLSOV26QMVJRDNGWSOKPT4MIGAMTHU6AIADGGJNG6526XAR3SF6Q, seg_pos={0 0}, upload=19402]: Done uploading piece. err: <nil> Aug 9 12:00:25.545685641 (18h34m19.976765772s) [noise=true , address=104.36.82.60:4354 , piece_id=HR52C2IWZNCL2XI553EUKTS37R5N4ILVW34GJSPWJV3Z3W7KKGWA, seg_pos={0 0}, upload=19402]: Done uploading piece. err: <nil> Aug 9 12:00:25.555951362 (18h34m19.987031493s) [noise=true , address=104.36.82.61:4396 , piece_id=LJ73GMLI4ZDNLDSGIRT4GRBU56YFUOBTOC7OW3NS4TRWSUXMQCHA, seg_pos={0 0}, upload=19402]: Done uploading piece. err: <nil> Aug 9 12:00:25.566816712 (18h34m19.997896843s) [noise=true , address=104.36.82.5:4845 , piece_id=5AJNZJD6MFKE3U6EZ27IEXQEO4KOUGSXA7EBCDSWVTVFYD4AJTHQ, seg_pos={0 0}, upload=19402]: Done uploading piece. err: <nil> Aug 9 12:00:25.585861129 (18h34m20.01694126s) [noise=true , address=104.36.82.45:4934 , piece_id=K7XWI37JOPKYII6BFSO6AZT73S3FYOH5JKCJXOM5O5SCWAG62OIQ, seg_pos={0 0}, upload=19402]: Done uploading piece. err: <nil> Aug 9 12:00:25.594196727 (18h34m20.025276858s) [noise=true , address=45.91.116.12:4243 , piece_id=FZW53SVI3PFJ3M6WQE74FPB737JEBGBGA2YB53XCU5DEPEUQHU7Q, seg_pos={0 0}, upload=19402]: Done uploading piece. err: <nil> Aug 9 12:00:25.606123457 (18h34m20.037203588s) [noise=true , address=77.90.31.12:4604 , piece_id=HPMFA765VZOJGNXPWE5SCA4AAK7QB4HFGUSYFQWOFLVZTU6PSUMA, seg_pos={0 0}, upload=19402]: Done uploading piece. err: <nil> Aug 9 12:00:25.606140419 (18h34m20.03722055s) [seg_pos={0 0}, upload=19402]: Segment reached optimal threshold of 14 pieces. Aug 9 12:00:25.606241462 (18h34m20.037321593s) [noise=true , address=104.36.82.13:4197 , piece_id=NEMRRYQPHXM73GSZVEETU7UNMUDQVNKKSLKQTRSBHBKPQ35KLLPQ, seg_pos={0 0}, upload=19402]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1cB89C5xnya9WGD9dGBsSouqkZH3nLqytrg9Qo4UH2kdFonTjM): protocol: expected piece hash; context canceled; EOF Aug 9 12:00:25.606346553 (18h34m20.037426675s) [noise=true , address=104.36.82.14:4242 , piece_id=2JREAT5B4ZOSHBJPMPKVQOW2VDMLQIDURO7KTYRPFJAM6U5K57WQ, seg_pos={0 0}, upload=19402]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1inR8rjUp4fRQWcaD2S7MRxveHyFxKoutWoEiuKQG3C895PaMz): protocol: expected piece hash; context canceled; EOF Aug 9 12:00:25.606405596 (18h34m20.037485727s) [noise=true , address=104.36.82.23:4454 , piece_id=VNKC653W6NNZTTH6CRDK2LSD5XFLLGJGYOOIELABDCKC364DITDA, seg_pos={0 0}, upload=19402]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1UCkuYGdVYLoR8oc1jpCZCN2smNfVqFwC8a7q95fJb8haB3MY2): protocol: expected piece hash; context canceled; EOF Aug 9 12:00:25.606463838 (18h34m20.037543969s) [noise=true , address=104.36.82.59:4327 , piece_id=WJJFOV44IRLH2E6BFC46ALBJXD54ML4HQOFSUDFNJ3OJUXASSTZQ, seg_pos={0 0}, upload=19402]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1V4C6AWGbyYQWfinoSj7iKovHeTJ9hqypyC5U9rZ8pveXBBfCY): protocol: expected piece hash; context canceled; EOF Aug 9 12:00:25.606478186 (18h34m20.037558317s) [seg_pos={0 0}, upload=19402]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 9 12:00:25.606535966 (18h34m20.037616097s) [upload=19402]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 9 12:00:25.606540855 (18h34m20.037620976s) [upload=19402]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 9 12:00:25.913625167 (18h34m20.344705298s) [upload=19402]: Done uploading segments... Aug 9 12:00:25.913644565 (18h34m20.344724696s) [upload=19402]: Upload finished. err: <nil> Duration: 5.869574006s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................X....................................] duration: 5.734124259s FAIL (2025-08-09 12:04:24 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 6) ======LOGS====== ADHZNIB6LBP5VQHFEGSYM3YNNZWNM2N4UU7JTCQ, seg_pos={0 0}, upload=18998]: Uploading piece... Aug 9 11:37:25.005421531 (18h11m19.436501662s) [noise=true , address=45.91.116.4:4940 , piece_id=M33ZGUCEF23HDGQZE5F5KLWQ43NHC7E7LKOVVGQ6P6TM2HFGB4WQ, seg_pos={0 0}, upload=18998]: Uploading piece... Aug 9 11:37:25.005614329 (18h11m19.43669446s) [noise=true , address=45.91.116.8:4047 , piece_id=B27B2QKZXICP4DHTDTCYSZZ7RN2PPCMKSRQBKRK5JJ4BE4RNWQAQ, seg_pos={0 0}, upload=18998]: Uploading piece... Aug 9 11:37:25.005727536 (18h11m19.436807667s) [noise=true , address=45.91.116.12:4225 , piece_id=VFK2SMVTBT76D7YLQWGRA6TF47JD3B7J7F5NFR4E2MQVNGNHYTAQ, seg_pos={0 0}, upload=18998]: Uploading piece... Aug 9 11:37:25.006281517 (18h11m19.437361648s) [noise=true , address=104.36.82.7:4909 , piece_id=OGG5ZCXSLIM5YKMM7K6KLKLWGOTX376DUTDT3567OB3AJIBPBKDQ, seg_pos={0 0}, upload=18998]: Uploading piece... Aug 9 11:37:25.006632558 (18h11m19.437712679s) [noise=true , address=45.91.116.13:4155 , piece_id=K4CGD4OCR2B6ZWAZHB5MDBECVJB3AC2HDXS2TUVAVGIWQYJEARLA, seg_pos={0 0}, upload=18998]: Uploading piece... Aug 9 11:37:25.006996264 (18h11m19.438076395s) [noise=true , address=77.90.31.18:4806 , piece_id=QCURQYEJL22LBT2YFBR62RIHH7VNSGL7GERP66GKHXK3JP5WLVZQ, seg_pos={0 0}, upload=18998]: Uploading piece... Aug 9 11:37:25.007245231 (18h11m19.438325362s) [noise=true , address=104.36.82.3:4765 , piece_id=HBLTTPZYK5ZXNG2XV53QVJA3NPX3YWKEAAAFVMDXUO6BLR3QKCWQ, seg_pos={0 0}, upload=18998]: Uploading piece... Aug 9 11:37:25.007475481 (18h11m19.438555612s) [noise=true , address=45.91.116.6:4973 , piece_id=MHDNXTVKFMLRKGS7JIPL4U5VQMBXGP4EYM7JEGPSK4U3GIJ5JEHQ, seg_pos={0 0}, upload=18998]: Uploading piece... Aug 9 11:37:25.007873984 (18h11m19.438954115s) [noise=true , address=45.91.116.7:4010 , piece_id=5G3O552GD5XO7SNIHST623QJBWJF2C7B6EN7ISGHHY3FYV6QL2DQ, seg_pos={0 0}, upload=18998]: Uploading piece... Aug 9 11:37:26.733124951 (18h11m21.164205082s) [upload=18998]: Next returned nil segment Aug 9 11:37:26.733142194 (18h11m21.164222325s) [upload=18998]: Waiting for error group managing segments... Aug 9 11:37:27.100757697 (18h11m21.531837848s) [noise=true , address=104.36.82.14:4222 , piece_id=Z4X6RJ5HBAYGUNKT6DXWSS2TEG2OUFPMH3NWYGA7D26XIFH46UAA, seg_pos={0 0}, upload=18998]: Done uploading piece. err: <nil> Aug 9 11:37:27.106781913 (18h11m21.537862064s) [noise=true , address=104.36.82.7:4912 , piece_id=MRA5X6N6MDQ2TADHZNIB6LBP5VQHFEGSYM3YNNZWNM2N4UU7JTCQ, seg_pos={0 0}, upload=18998]: Done uploading piece. err: <nil> Aug 9 11:37:27.136679105 (18h11m21.567759236s) [noise=true , address=104.36.82.5:4838 , piece_id=T3WCWCLB4WAXQY3S6VCFHLKYK6IY2Q6P2776CLGYS4QOFTXMYVEQ, seg_pos={0 0}, upload=18998]: Done uploading piece. err: <nil> Aug 9 11:37:27.139126447 (18h11m21.570206578s) [noise=true , address=104.36.82.3:4765 , piece_id=HBLTTPZYK5ZXNG2XV53QVJA3NPX3YWKEAAAFVMDXUO6BLR3QKCWQ, seg_pos={0 0}, upload=18998]: Done uploading piece. err: <nil> Aug 9 11:37:27.183514474 (18h11m21.614594606s) [noise=true , address=104.36.82.7:4909 , piece_id=OGG5ZCXSLIM5YKMM7K6KLKLWGOTX376DUTDT3567OB3AJIBPBKDQ, seg_pos={0 0}, upload=18998]: Done uploading piece. err: <nil> Aug 9 11:37:53.489259032 (18h11m47.920339163s) [noise=true , address=45.91.116.13:4155 , piece_id=K4CGD4OCR2B6ZWAZHB5MDBECVJB3AC2HDXS2TUVAVGIWQYJEARLA, seg_pos={0 0}, upload=18998]: Done uploading piece. err: <nil> Aug 9 11:37:54.124525138 (18h11m48.555605259s) [noise=true , address=45.91.116.7:4010 , piece_id=5G3O552GD5XO7SNIHST623QJBWJF2C7B6EN7ISGHHY3FYV6QL2DQ, seg_pos={0 0}, upload=18998]: Done uploading piece. err: ecclient: upload failed (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w, address:45.91.116.7:4010): protocol: write tcp 188.166.225.170:54960->45.91.116.7:4010: use of closed network connection; write tcp 188.166.225.170:54960->45.91.116.7:4010: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54960->45.91.116.7:4010: use of closed network connection Aug 9 11:37:54.124581035 (18h11m48.555661166s) [noise=true , address=77.90.31.10:4520 , piece_id=YRBBWM54FGU3B74JQ7EOAO7VSGVFXAVT4SNZ6KN3IQFIHIEVV5HA, seg_pos={0 0}, upload=18998]: Done uploading piece. err: ecclient: upload failed (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp, address:77.90.31.10:4520): protocol: write tcp 188.166.225.170:39560->77.90.31.10:4520: use of closed network connection; write tcp 188.166.225.170:39560->77.90.31.10:4520: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39560->77.90.31.10:4520: use of closed network connection Aug 9 11:37:54.124615480 (18h11m48.555695652s) [noise=true , address=45.91.116.12:4225 , piece_id=VFK2SMVTBT76D7YLQWGRA6TF47JD3B7J7F5NFR4E2MQVNGNHYTAQ, seg_pos={0 0}, upload=18998]: Done uploading piece. err: ecclient: upload failed (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy, address:45.91.116.12:4225): protocol: write tcp 188.166.225.170:38054->45.91.116.12:4225: use of closed network connection; write tcp 188.166.225.170:38054->45.91.116.12:4225: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38054->45.91.116.12:4225: use of closed network connection Aug 9 11:37:54.124643875 (18h11m48.555724016s) [noise=true , address=45.91.116.4:4940 , piece_id=M33ZGUCEF23HDGQZE5F5KLWQ43NHC7E7LKOVVGQ6P6TM2HFGB4WQ, seg_pos={0 0}, upload=18998]: Done uploading piece. err: ecclient: upload failed (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe, address:45.91.116.4:4940): protocol: write tcp 188.166.225.170:40630->45.91.116.4:4940: use of closed network connection; write tcp 188.166.225.170:40630->45.91.116.4:4940: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40630->45.91.116.4:4940: use of closed network connection Aug 9 11:37:54.124693159 (18h11m48.55577335s) [noise=true , address=45.91.116.14:4189 , piece_id=XVQMTYHB6H3IKSERYT2BEB4NP42KV6FRE6J2MKKU5WEOCVBHYIWA, seg_pos={0 0}, upload=18998]: Done uploading piece. err: ecclient: upload failed (node:1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N, address:45.91.116.14:4189): protocol: write tcp 188.166.225.170:37650->45.91.116.14:4189: use of closed network connection; write tcp 188.166.225.170:37650->45.91.116.14:4189: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37650->45.91.116.14:4189: use of closed network connection Aug 9 11:37:54.124771740 (18h11m48.555851871s) [noise=true , address=77.90.31.18:4806 , piece_id=QCURQYEJL22LBT2YFBR62RIHH7VNSGL7GERP66GKHXK3JP5WLVZQ, seg_pos={0 0}, upload=18998]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: write tcp 188.166.225.170:55536->77.90.31.18:4806: use of closed network connection; write tcp 188.166.225.170:55536->77.90.31.18:4806: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55536->77.90.31.18:4806: use of closed network connection Aug 9 11:37:54.124817217 (18h11m48.555897348s) [noise=true , address=77.90.31.15:4697 , piece_id=Q26SRGZO2YBJWPDIG4U422EICE6XEXPLHFHXLKFUBKMMJAZ4W7UQ, seg_pos={0 0}, upload=18998]: Done uploading piece. err: ecclient: upload failed (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, address:77.90.31.15:4697): protocol: write tcp 188.166.225.170:49618->77.90.31.15:4697: use of closed network connection; write tcp 188.166.225.170:49618->77.90.31.15:4697: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49618->77.90.31.15:4697: use of closed network connection Aug 9 11:37:54.124833387 (18h11m48.555913559s) [noise=true , address=45.91.116.6:4973 , piece_id=MHDNXTVKFMLRKGS7JIPL4U5VQMBXGP4EYM7JEGPSK4U3GIJ5JEHQ, seg_pos={0 0}, upload=18998]: Done uploading piece. err: ecclient: upload failed (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, address:45.91.116.6:4973): protocol: write tcp 188.166.225.170:59438->45.91.116.6:4973: use of closed network connection; write tcp 188.166.225.170:59438->45.91.116.6:4973: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59438->45.91.116.6:4973: use of closed network connection Aug 9 11:37:54.124840752 (18h11m48.555920883s) [noise=true , address=77.90.31.17:4772 , piece_id=MGZPHOODRIOTGABQJHNIP5FGF4XN6A723ZHUBKI6KNAJDKFWWOJA, seg_pos={0 0}, upload=18998]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: write tcp 188.166.225.170:42704->77.90.31.17:4772: use of closed network connection; write tcp 188.166.225.170:42704->77.90.31.17:4772: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42704->77.90.31.17:4772: use of closed network connection Aug 9 11:37:54.124845251 (18h11m48.555925382s) [noise=true , address=45.91.116.8:4047 , piece_id=B27B2QKZXICP4DHTDTCYSZZ7RN2PPCMKSRQBKRK5JJ4BE4RNWQAQ, seg_pos={0 0}, upload=18998]: Done uploading piece. err: ecclient: upload failed (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN, address:45.91.116.8:4047): protocol: write tcp 188.166.225.170:40794->45.91.116.8:4047: use of closed network connection; write tcp 188.166.225.170:40794->45.91.116.8:4047: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40794->45.91.116.8:4047: use of closed network connection Aug 9 11:37:54.124866321 (18h11m48.555946492s) [seg_pos={0 0}, upload=18998]: Done waiting for segment. successful: 6 optimal: 13 errs: <nil> Aug 9 11:37:54.353990811 (18h11m48.785070942s) [upload=18998]: Done uploading segments... Aug 9 11:37:54.354028673 (18h11m48.785108804s) [upload=18998]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 6) Duration: 5.734153576s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 3 out of 60 is failed (threshold=3) [........................X..................X.........X......] duration: 2.069199326s FAIL (2025-08-09 12:04:22 UTC) orbiter warning: operation: ListObjects ======ERROR====== unable to find key in bucket: sj://uptime-checks-placement-20/orbiter_us_0/1K ======LOGS====== Aug 9 11:59:31.819436107 (18h33m26.250516238s) [upload=19388]: Starting upload Aug 9 11:59:31.819492355 (18h33m26.250572516s) [upload=19388]: Done starting upload Aug 9 11:59:31.819563080 (18h33m26.250643211s) [upload=19388]: Uploading segments... Aug 9 11:59:31.819616853 (18h33m26.250696984s) [upload=19388]: Got next segment. Inline: true Aug 9 11:59:31.819621552 (18h33m26.250701683s) [upload=19388]: Waiting for error group managing segments... Aug 9 11:59:31.819660466 (18h33m26.250740597s) [upload=19388]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 9 11:59:31.819665206 (18h33m26.250745327s) [upload=19388]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 9 11:59:31.819667851 (18h33m26.250747972s) [upload=19388]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 9 11:59:32.242597809 (18h33m26.67367794s) [upload=19388]: Done uploading segments... Aug 9 11:59:32.242615142 (18h33m26.673695263s) [upload=19388]: Upload finished. err: <nil> Duration:* 2.069223942s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................X...................................] duration: 5.716001662s FAIL (2025-08-09 12:04:02 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 6) ======LOGS====== ADHZNIB6LBP5VQHFEGSYM3YNNZWNM2N4UU7JTCQ, seg_pos={0 0}, upload=18998]: Uploading piece... Aug 9 11:37:25.005421531 (18h11m19.436501662s) [noise=true , address=45.91.116.4:4940 , piece_id=M33ZGUCEF23HDGQZE5F5KLWQ43NHC7E7LKOVVGQ6P6TM2HFGB4WQ, seg_pos={0 0}, upload=18998]: Uploading piece... Aug 9 11:37:25.005614329 (18h11m19.43669446s) [noise=true , address=45.91.116.8:4047 , piece_id=B27B2QKZXICP4DHTDTCYSZZ7RN2PPCMKSRQBKRK5JJ4BE4RNWQAQ, seg_pos={0 0}, upload=18998]: Uploading piece... Aug 9 11:37:25.005727536 (18h11m19.436807667s) [noise=true , address=45.91.116.12:4225 , piece_id=VFK2SMVTBT76D7YLQWGRA6TF47JD3B7J7F5NFR4E2MQVNGNHYTAQ, seg_pos={0 0}, upload=18998]: Uploading piece... Aug 9 11:37:25.006281517 (18h11m19.437361648s) [noise=true , address=104.36.82.7:4909 , piece_id=OGG5ZCXSLIM5YKMM7K6KLKLWGOTX376DUTDT3567OB3AJIBPBKDQ, seg_pos={0 0}, upload=18998]: Uploading piece... Aug 9 11:37:25.006632558 (18h11m19.437712679s) [noise=true , address=45.91.116.13:4155 , piece_id=K4CGD4OCR2B6ZWAZHB5MDBECVJB3AC2HDXS2TUVAVGIWQYJEARLA, seg_pos={0 0}, upload=18998]: Uploading piece... Aug 9 11:37:25.006996264 (18h11m19.438076395s) [noise=true , address=77.90.31.18:4806 , piece_id=QCURQYEJL22LBT2YFBR62RIHH7VNSGL7GERP66GKHXK3JP5WLVZQ, seg_pos={0 0}, upload=18998]: Uploading piece... Aug 9 11:37:25.007245231 (18h11m19.438325362s) [noise=true , address=104.36.82.3:4765 , piece_id=HBLTTPZYK5ZXNG2XV53QVJA3NPX3YWKEAAAFVMDXUO6BLR3QKCWQ, seg_pos={0 0}, upload=18998]: Uploading piece... Aug 9 11:37:25.007475481 (18h11m19.438555612s) [noise=true , address=45.91.116.6:4973 , piece_id=MHDNXTVKFMLRKGS7JIPL4U5VQMBXGP4EYM7JEGPSK4U3GIJ5JEHQ, seg_pos={0 0}, upload=18998]: Uploading piece... Aug 9 11:37:25.007873984 (18h11m19.438954115s) [noise=true , address=45.91.116.7:4010 , piece_id=5G3O552GD5XO7SNIHST623QJBWJF2C7B6EN7ISGHHY3FYV6QL2DQ, seg_pos={0 0}, upload=18998]: Uploading piece... Aug 9 11:37:26.733124951 (18h11m21.164205082s) [upload=18998]: Next returned nil segment Aug 9 11:37:26.733142194 (18h11m21.164222325s) [upload=18998]: Waiting for error group managing segments... Aug 9 11:37:27.100757697 (18h11m21.531837848s) [noise=true , address=104.36.82.14:4222 , piece_id=Z4X6RJ5HBAYGUNKT6DXWSS2TEG2OUFPMH3NWYGA7D26XIFH46UAA, seg_pos={0 0}, upload=18998]: Done uploading piece. err: <nil> Aug 9 11:37:27.106781913 (18h11m21.537862064s) [noise=true , address=104.36.82.7:4912 , piece_id=MRA5X6N6MDQ2TADHZNIB6LBP5VQHFEGSYM3YNNZWNM2N4UU7JTCQ, seg_pos={0 0}, upload=18998]: Done uploading piece. err: <nil> Aug 9 11:37:27.136679105 (18h11m21.567759236s) [noise=true , address=104.36.82.5:4838 , piece_id=T3WCWCLB4WAXQY3S6VCFHLKYK6IY2Q6P2776CLGYS4QOFTXMYVEQ, seg_pos={0 0}, upload=18998]: Done uploading piece. err: <nil> Aug 9 11:37:27.139126447 (18h11m21.570206578s) [noise=true , address=104.36.82.3:4765 , piece_id=HBLTTPZYK5ZXNG2XV53QVJA3NPX3YWKEAAAFVMDXUO6BLR3QKCWQ, seg_pos={0 0}, upload=18998]: Done uploading piece. err: <nil> Aug 9 11:37:27.183514474 (18h11m21.614594606s) [noise=true , address=104.36.82.7:4909 , piece_id=OGG5ZCXSLIM5YKMM7K6KLKLWGOTX376DUTDT3567OB3AJIBPBKDQ, seg_pos={0 0}, upload=18998]: Done uploading piece. err: <nil> Aug 9 11:37:53.489259032 (18h11m47.920339163s) [noise=true , address=45.91.116.13:4155 , piece_id=K4CGD4OCR2B6ZWAZHB5MDBECVJB3AC2HDXS2TUVAVGIWQYJEARLA, seg_pos={0 0}, upload=18998]: Done uploading piece. err: <nil> Aug 9 11:37:54.124525138 (18h11m48.555605259s) [noise=true , address=45.91.116.7:4010 , piece_id=5G3O552GD5XO7SNIHST623QJBWJF2C7B6EN7ISGHHY3FYV6QL2DQ, seg_pos={0 0}, upload=18998]: Done uploading piece. err: ecclient: upload failed (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w, address:45.91.116.7:4010): protocol: write tcp 188.166.225.170:54960->45.91.116.7:4010: use of closed network connection; write tcp 188.166.225.170:54960->45.91.116.7:4010: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54960->45.91.116.7:4010: use of closed network connection Aug 9 11:37:54.124581035 (18h11m48.555661166s) [noise=true , address=77.90.31.10:4520 , piece_id=YRBBWM54FGU3B74JQ7EOAO7VSGVFXAVT4SNZ6KN3IQFIHIEVV5HA, seg_pos={0 0}, upload=18998]: Done uploading piece. err: ecclient: upload failed (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp, address:77.90.31.10:4520): protocol: write tcp 188.166.225.170:39560->77.90.31.10:4520: use of closed network connection; write tcp 188.166.225.170:39560->77.90.31.10:4520: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39560->77.90.31.10:4520: use of closed network connection Aug 9 11:37:54.124615480 (18h11m48.555695652s) [noise=true , address=45.91.116.12:4225 , piece_id=VFK2SMVTBT76D7YLQWGRA6TF47JD3B7J7F5NFR4E2MQVNGNHYTAQ, seg_pos={0 0}, upload=18998]: Done uploading piece. err: ecclient: upload failed (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy, address:45.91.116.12:4225): protocol: write tcp 188.166.225.170:38054->45.91.116.12:4225: use of closed network connection; write tcp 188.166.225.170:38054->45.91.116.12:4225: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38054->45.91.116.12:4225: use of closed network connection Aug 9 11:37:54.124643875 (18h11m48.555724016s) [noise=true , address=45.91.116.4:4940 , piece_id=M33ZGUCEF23HDGQZE5F5KLWQ43NHC7E7LKOVVGQ6P6TM2HFGB4WQ, seg_pos={0 0}, upload=18998]: Done uploading piece. err: ecclient: upload failed (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe, address:45.91.116.4:4940): protocol: write tcp 188.166.225.170:40630->45.91.116.4:4940: use of closed network connection; write tcp 188.166.225.170:40630->45.91.116.4:4940: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40630->45.91.116.4:4940: use of closed network connection Aug 9 11:37:54.124693159 (18h11m48.55577335s) [noise=true , address=45.91.116.14:4189 , piece_id=XVQMTYHB6H3IKSERYT2BEB4NP42KV6FRE6J2MKKU5WEOCVBHYIWA, seg_pos={0 0}, upload=18998]: Done uploading piece. err: ecclient: upload failed (node:1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N, address:45.91.116.14:4189): protocol: write tcp 188.166.225.170:37650->45.91.116.14:4189: use of closed network connection; write tcp 188.166.225.170:37650->45.91.116.14:4189: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37650->45.91.116.14:4189: use of closed network connection Aug 9 11:37:54.124771740 (18h11m48.555851871s) [noise=true , address=77.90.31.18:4806 , piece_id=QCURQYEJL22LBT2YFBR62RIHH7VNSGL7GERP66GKHXK3JP5WLVZQ, seg_pos={0 0}, upload=18998]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: write tcp 188.166.225.170:55536->77.90.31.18:4806: use of closed network connection; write tcp 188.166.225.170:55536->77.90.31.18:4806: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55536->77.90.31.18:4806: use of closed network connection Aug 9 11:37:54.124817217 (18h11m48.555897348s) [noise=true , address=77.90.31.15:4697 , piece_id=Q26SRGZO2YBJWPDIG4U422EICE6XEXPLHFHXLKFUBKMMJAZ4W7UQ, seg_pos={0 0}, upload=18998]: Done uploading piece. err: ecclient: upload failed (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, address:77.90.31.15:4697): protocol: write tcp 188.166.225.170:49618->77.90.31.15:4697: use of closed network connection; write tcp 188.166.225.170:49618->77.90.31.15:4697: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49618->77.90.31.15:4697: use of closed network connection Aug 9 11:37:54.124833387 (18h11m48.555913559s) [noise=true , address=45.91.116.6:4973 , piece_id=MHDNXTVKFMLRKGS7JIPL4U5VQMBXGP4EYM7JEGPSK4U3GIJ5JEHQ, seg_pos={0 0}, upload=18998]: Done uploading piece. err: ecclient: upload failed (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, address:45.91.116.6:4973): protocol: write tcp 188.166.225.170:59438->45.91.116.6:4973: use of closed network connection; write tcp 188.166.225.170:59438->45.91.116.6:4973: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59438->45.91.116.6:4973: use of closed network connection Aug 9 11:37:54.124840752 (18h11m48.555920883s) [noise=true , address=77.90.31.17:4772 , piece_id=MGZPHOODRIOTGABQJHNIP5FGF4XN6A723ZHUBKI6KNAJDKFWWOJA, seg_pos={0 0}, upload=18998]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: write tcp 188.166.225.170:42704->77.90.31.17:4772: use of closed network connection; write tcp 188.166.225.170:42704->77.90.31.17:4772: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42704->77.90.31.17:4772: use of closed network connection Aug 9 11:37:54.124845251 (18h11m48.555925382s) [noise=true , address=45.91.116.8:4047 , piece_id=B27B2QKZXICP4DHTDTCYSZZ7RN2PPCMKSRQBKRK5JJ4BE4RNWQAQ, seg_pos={0 0}, upload=18998]: Done uploading piece. err: ecclient: upload failed (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN, address:45.91.116.8:4047): protocol: write tcp 188.166.225.170:40794->45.91.116.8:4047: use of closed network connection; write tcp 188.166.225.170:40794->45.91.116.8:4047: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40794->45.91.116.8:4047: use of closed network connection Aug 9 11:37:54.124866321 (18h11m48.555946492s) [seg_pos={0 0}, upload=18998]: Done waiting for segment. successful: 6 optimal: 13 errs: <nil> Aug 9 11:37:54.353990811 (18h11m48.785070942s) [upload=18998]: Done uploading segments... Aug 9 11:37:54.354028673 (18h11m48.785108804s) [upload=18998]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 6) Duration: 5.716029745s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 3 out of 60 is failed (threshold=3) [......................................X............X.X......] duration: 5.823009981s FAIL (2025-08-09 12:04:01 UTC) orbiter warning: operation: ListObjects ======ERROR====== unable to find key in bucket: sj://uptime-checks-placement-20/orbiter_us_0/4M ======LOGS====== Aug 9 12:00:23.108483377 (18h34m17.539563508s) [upload=19402]: Starting upload Aug 9 12:00:23.108636078 (18h34m17.53971621s) [upload=19402]: Done starting upload Aug 9 12:00:23.108746840 (18h34m17.539826981s) [upload=19402]: Uploading segments... Aug 9 12:00:23.108904442 (18h34m17.539984573s) [upload=19402]: Got next segment. Inline: false Aug 9 12:00:23.108970418 (18h34m17.540050599s) [upload=19402]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 9 12:00:23.108977743 (18h34m17.540057904s) [upload=19402]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 9 12:00:23.579075368 (18h34m18.010155499s) [seg_pos={0 0}, upload=19402]: Begin upload segment... Aug 9 12:00:23.579127947 (18h34m18.010208078s) [seg_pos={0 0}, upload=19402]: Done begin upload segment. Aug 9 12:00:23.579157074 (18h34m18.010237205s) [noise=true , address=104.36.82.44:4899 , piece_id=KIQ5AFZ2FIUJ3XIG5ZQBJYITK5CD4ZDW5SYOUPFIYAOZBPENWFFA, seg_pos={0 0}, upload=19402]: Uploading piece... Aug 9 12:00:23.579206939 (18h34m18.01028707s) [noise=true , address=104.36.82.55:4199 , piece_id=6D4WT6KDBSMMCPXONV74YG4IXFEXPXU2FQTCL2W2NPZ2ERVMBUMQ, seg_pos={0 0}, upload=19402]: Uploading piece... Aug 9 12:00:23.579552951 (18h34m18.010633082s) [noise=true , address=64.49.13.139:4469 , piece_id=UTZT7WZ3LQ6LOZ3C6IXV5M7L3APATJOYDRZAMWUEYCZP4I34THCA, seg_pos={0 0}, upload=19402]: Uploading piece... Aug 9 12:00:23.579921666 (18h34m18.011001797s) [noise=true , address=104.36.82.59:4327 , piece_id=WJJFOV44IRLH2E6BFC46ALBJXD54ML4HQOFSUDFNJ3OJUXASSTZQ, seg_pos={0 0}, upload=19402]: Uploading piece... Aug 9 12:00:23.580275103 (18h34m18.011355234s) [noise=true , address=104.36.82.13:4197 , piece_id=NEMRRYQPHXM73GSZVEETU7UNMUDQVNKKSLKQTRSBHBKPQ35KLLPQ, seg_pos={0 0}, upload=19402]: Uploading piece... Aug 9 12:00:23.580699144 (18h34m18.011779275s) [noise=true , address=104.36.82.23:4454 , piece_id=VNKC653W6NNZTTH6CRDK2LSD5XFLLGJGYOOIELABDCKC364DITDA, seg_pos={0 0}, upload=19402]: Uploading piece... Aug 9 12:00:23.580803934 (18h34m18.011884065s) [noise=true , address=104.36.82.61:4396 , piece_id=LJ73GMLI4ZDNLDSGIRT4GRBU56YFUOBTOC7OW3NS4TRWSUXMQCHA, seg_pos={0 0}, upload=19402]: Uploading piece... Aug 9 12:00:23.581071286 (18h34m18.012151417s) [noise=true , address=104.36.82.45:4934 , piece_id=K7XWI37JOPKYII6BFSO6AZT73S3FYOH5JKCJXOM5O5SCWAG62OIQ, seg_pos={0 0}, upload=19402]: Uploading piece... Aug 9 12:00:23.581305595 (18h34m18.012385726s) [noise=true , address=104.36.82.60:4354 , piece_id=HR52C2IWZNCL2XI553EUKTS37R5N4ILVW34GJSPWJV3Z3W7KKGWA, seg_pos={0 0}, upload=19402]: Uploading piece... Aug 9 12:00:23.581541305 (18h34m18.012621436s) [noise=true , address=45.91.116.12:4243 , piece_id=FZW53SVI3PFJ3M6WQE74FPB737JEBGBGA2YB53XCU5DEPEUQHU7Q, seg_pos={0 0}, upload=19402]: Uploading piece... Aug 9 12:00:23.581620086 (18h34m18.012700217s) [noise=true , address=104.36.82.58:4268 , piece_id=QLMKG6PZ4XYJJUTOHFW7EWTFGTST5HYLTHP5MMX6X3AJJJAZSZ3Q, seg_pos={0 0}, upload=19402]: Uploading piece... Aug 9 12:00:23.581922384 (18h34m18.013002515s) [noise=true , address=104.36.82.9:4031 , piece_id=LLSOV26QMVJRDNGWSOKPT4MIGAMTHU6AIADGGJNG6526XAR3SF6Q, seg_pos={0 0}, upload=19402]: Uploading piece... Aug 9 12:00:23.582143177 (18h34m18.013223308s) [noise=true , address=104.36.82.60:4358 , piece_id=WPW7FORNXWY6DOAXVH775D4W3VFOH7PRMQO5UXKJNIFFCRVSE43A, seg_pos={0 0}, upload=19402]: Uploading piece... Aug 9 12:00:23.582347538 (18h34m18.013427669s) [noise=true , address=104.36.82.14:4242 , piece_id=2JREAT5B4ZOSHBJPMPKVQOW2VDMLQIDURO7KTYRPFJAM6U5K57WQ, seg_pos={0 0}, upload=19402]: Uploading piece... Aug 9 12:00:23.582596004 (18h34m18.013676135s) [noise=true , address=64.49.13.170:4726 , piece_id=A2OWN4ZBVO3MZKLRAHERFMYAZDSLB4HMYESALCEVEH66LGUZEQOQ, seg_pos={0 0}, upload=19402]: Uploading piece... Aug 9 12:00:23.582774395 (18h34m18.013854526s) [noise=true , address=104.36.82.46:4484 , piece_id=FQHBXDSLXK2GIOSY4V5QBBPLOH7ZD3KHJVXR5IFYPMICQCWH7UFA, seg_pos={0 0}, upload=19402]: Uploading piece... Aug 9 12:00:23.583063899 (18h34m18.01414403s) [noise=true , address=77.90.31.12:4604 , piece_id=HPMFA765VZOJGNXPWE5SCA4AAK7QB4HFGUSYFQWOFLVZTU6PSUMA, seg_pos={0 0}, upload=19402]: Uploading piece... Aug 9 12:00:23.583138481 (18h34m18.014218612s) [noise=true , address=104.36.82.5:4845 , piece_id=5AJNZJD6MFKE3U6EZ27IEXQEO4KOUGSXA7EBCDSWVTVFYD4AJTHQ, seg_pos={0 0}, upload=19402]: Uploading piece... Aug 9 12:00:24.683477824 (18h34m19.114557955s) [upload=19402]: Next returned nil segment Aug 9 12:00:24.683489506 (18h34m19.114569637s) [upload=19402]: Waiting for error group managing segments... Aug 9 12:00:24.955322603 (18h34m19.386402734s) [noise=true , address=64.49.13.170:4726 , piece_id=A2OWN4ZBVO3MZKLRAHERFMYAZDSLB4HMYESALCEVEH66LGUZEQOQ, seg_pos={0 0}, upload=19402]: Done uploading piece. err: <nil> Aug 9 12:00:24.969832404 (18h34m19.400912535s) [noise=true , address=64.49.13.139:4469 , piece_id=UTZT7WZ3LQ6LOZ3C6IXV5M7L3APATJOYDRZAMWUEYCZP4I34THCA, seg_pos={0 0}, upload=19402]: Done uploading piece. err: <nil> Aug 9 12:00:25.516280300 (18h34m19.947360431s) [noise=true , address=104.36.82.55:4199 , piece_id=6D4WT6KDBSMMCPXONV74YG4IXFEXPXU2FQTCL2W2NPZ2ERVMBUMQ, seg_pos={0 0}, upload=19402]: Done uploading piece. err: <nil> Aug 9 12:00:25.525386162 (18h34m19.956466293s) [noise=true , address=104.36.82.46:4484 , piece_id=FQHBXDSLXK2GIOSY4V5QBBPLOH7ZD3KHJVXR5IFYPMICQCWH7UFA, seg_pos={0 0}, upload=19402]: Done uploading piece. err: <nil> Aug 9 12:00:25.526709204 (18h34m19.957789335s) [noise=true , address=104.36.82.60:4358 , piece_id=WPW7FORNXWY6DOAXVH775D4W3VFOH7PRMQO5UXKJNIFFCRVSE43A, seg_pos={0 0}, upload=19402]: Done uploading piece. err: <nil> Aug 9 12:00:25.530790893 (18h34m19.961871024s) [noise=true , address=104.36.82.44:4899 , piece_id=KIQ5AFZ2FIUJ3XIG5ZQBJYITK5CD4ZDW5SYOUPFIYAOZBPENWFFA, seg_pos={0 0}, upload=19402]: Done uploading piece. err: <nil> Aug 9 12:00:25.531900687 (18h34m19.962980818s) [noise=true , address=104.36.82.58:4268 , piece_id=QLMKG6PZ4XYJJUTOHFW7EWTFGTST5HYLTHP5MMX6X3AJJJAZSZ3Q, seg_pos={0 0}, upload=19402]: Done uploading piece. err: <nil> Aug 9 12:00:25.536797967 (18h34m19.967878098s) [noise=true , address=104.36.82.9:4031 , piece_id=LLSOV26QMVJRDNGWSOKPT4MIGAMTHU6AIADGGJNG6526XAR3SF6Q, seg_pos={0 0}, upload=19402]: Done uploading piece. err: <nil> Aug 9 12:00:25.545685641 (18h34m19.976765772s) [noise=true , address=104.36.82.60:4354 , piece_id=HR52C2IWZNCL2XI553EUKTS37R5N4ILVW34GJSPWJV3Z3W7KKGWA, seg_pos={0 0}, upload=19402]: Done uploading piece. err: <nil> Aug 9 12:00:25.555951362 (18h34m19.987031493s) [noise=true , address=104.36.82.61:4396 , piece_id=LJ73GMLI4ZDNLDSGIRT4GRBU56YFUOBTOC7OW3NS4TRWSUXMQCHA, seg_pos={0 0}, upload=19402]: Done uploading piece. err: <nil> Aug 9 12:00:25.566816712 (18h34m19.997896843s) [noise=true , address=104.36.82.5:4845 , piece_id=5AJNZJD6MFKE3U6EZ27IEXQEO4KOUGSXA7EBCDSWVTVFYD4AJTHQ, seg_pos={0 0}, upload=19402]: Done uploading piece. err: <nil> Aug 9 12:00:25.585861129 (18h34m20.01694126s) [noise=true , address=104.36.82.45:4934 , piece_id=K7XWI37JOPKYII6BFSO6AZT73S3FYOH5JKCJXOM5O5SCWAG62OIQ, seg_pos={0 0}, upload=19402]: Done uploading piece. err: <nil> Aug 9 12:00:25.594196727 (18h34m20.025276858s) [noise=true , address=45.91.116.12:4243 , piece_id=FZW53SVI3PFJ3M6WQE74FPB737JEBGBGA2YB53XCU5DEPEUQHU7Q, seg_pos={0 0}, upload=19402]: Done uploading piece. err: <nil> Aug 9 12:00:25.606123457 (18h34m20.037203588s) [noise=true , address=77.90.31.12:4604 , piece_id=HPMFA765VZOJGNXPWE5SCA4AAK7QB4HFGUSYFQWOFLVZTU6PSUMA, seg_pos={0 0}, upload=19402]: Done uploading piece. err: <nil> Aug 9 12:00:25.606140419 (18h34m20.03722055s) [seg_pos={0 0}, upload=19402]: Segment reached optimal threshold of 14 pieces. Aug 9 12:00:25.606241462 (18h34m20.037321593s) [noise=true , address=104.36.82.13:4197 , piece_id=NEMRRYQPHXM73GSZVEETU7UNMUDQVNKKSLKQTRSBHBKPQ35KLLPQ, seg_pos={0 0}, upload=19402]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1cB89C5xnya9WGD9dGBsSouqkZH3nLqytrg9Qo4UH2kdFonTjM): protocol: expected piece hash; context canceled; EOF Aug 9 12:00:25.606346553 (18h34m20.037426675s) [noise=true , address=104.36.82.14:4242 , piece_id=2JREAT5B4ZOSHBJPMPKVQOW2VDMLQIDURO7KTYRPFJAM6U5K57WQ, seg_pos={0 0}, upload=19402]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1inR8rjUp4fRQWcaD2S7MRxveHyFxKoutWoEiuKQG3C895PaMz): protocol: expected piece hash; context canceled; EOF Aug 9 12:00:25.606405596 (18h34m20.037485727s) [noise=true , address=104.36.82.23:4454 , piece_id=VNKC653W6NNZTTH6CRDK2LSD5XFLLGJGYOOIELABDCKC364DITDA, seg_pos={0 0}, upload=19402]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1UCkuYGdVYLoR8oc1jpCZCN2smNfVqFwC8a7q95fJb8haB3MY2): protocol: expected piece hash; context canceled; EOF Aug 9 12:00:25.606463838 (18h34m20.037543969s) [noise=true , address=104.36.82.59:4327 , piece_id=WJJFOV44IRLH2E6BFC46ALBJXD54ML4HQOFSUDFNJ3OJUXASSTZQ, seg_pos={0 0}, upload=19402]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1V4C6AWGbyYQWfinoSj7iKovHeTJ9hqypyC5U9rZ8pveXBBfCY): protocol: expected piece hash; context canceled; EOF Aug 9 12:00:25.606478186 (18h34m20.037558317s) [seg_pos={0 0}, upload=19402]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 9 12:00:25.606535966 (18h34m20.037616097s) [upload=19402]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 9 12:00:25.606540855 (18h34m20.037620976s) [upload=19402]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 9 12:00:25.913625167 (18h34m20.344705298s) [upload=19402]: Done uploading segments... Aug 9 12:00:25.913644565 (18h34m20.344724696s) [upload=19402]: Upload finished. err: <nil> Duration: 5.823037323s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................X..................................] duration: 5.786243885s FAIL (2025-08-09 12:03:46 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 6) ======LOGS====== ADHZNIB6LBP5VQHFEGSYM3YNNZWNM2N4UU7JTCQ, seg_pos={0 0}, upload=18998]: Uploading piece... Aug 9 11:37:25.005421531 (18h11m19.436501662s) [noise=true , address=45.91.116.4:4940 , piece_id=M33ZGUCEF23HDGQZE5F5KLWQ43NHC7E7LKOVVGQ6P6TM2HFGB4WQ, seg_pos={0 0}, upload=18998]: Uploading piece... Aug 9 11:37:25.005614329 (18h11m19.43669446s) [noise=true , address=45.91.116.8:4047 , piece_id=B27B2QKZXICP4DHTDTCYSZZ7RN2PPCMKSRQBKRK5JJ4BE4RNWQAQ, seg_pos={0 0}, upload=18998]: Uploading piece... Aug 9 11:37:25.005727536 (18h11m19.436807667s) [noise=true , address=45.91.116.12:4225 , piece_id=VFK2SMVTBT76D7YLQWGRA6TF47JD3B7J7F5NFR4E2MQVNGNHYTAQ, seg_pos={0 0}, upload=18998]: Uploading piece... Aug 9 11:37:25.006281517 (18h11m19.437361648s) [noise=true , address=104.36.82.7:4909 , piece_id=OGG5ZCXSLIM5YKMM7K6KLKLWGOTX376DUTDT3567OB3AJIBPBKDQ, seg_pos={0 0}, upload=18998]: Uploading piece... Aug 9 11:37:25.006632558 (18h11m19.437712679s) [noise=true , address=45.91.116.13:4155 , piece_id=K4CGD4OCR2B6ZWAZHB5MDBECVJB3AC2HDXS2TUVAVGIWQYJEARLA, seg_pos={0 0}, upload=18998]: Uploading piece... Aug 9 11:37:25.006996264 (18h11m19.438076395s) [noise=true , address=77.90.31.18:4806 , piece_id=QCURQYEJL22LBT2YFBR62RIHH7VNSGL7GERP66GKHXK3JP5WLVZQ, seg_pos={0 0}, upload=18998]: Uploading piece... Aug 9 11:37:25.007245231 (18h11m19.438325362s) [noise=true , address=104.36.82.3:4765 , piece_id=HBLTTPZYK5ZXNG2XV53QVJA3NPX3YWKEAAAFVMDXUO6BLR3QKCWQ, seg_pos={0 0}, upload=18998]: Uploading piece... Aug 9 11:37:25.007475481 (18h11m19.438555612s) [noise=true , address=45.91.116.6:4973 , piece_id=MHDNXTVKFMLRKGS7JIPL4U5VQMBXGP4EYM7JEGPSK4U3GIJ5JEHQ, seg_pos={0 0}, upload=18998]: Uploading piece... Aug 9 11:37:25.007873984 (18h11m19.438954115s) [noise=true , address=45.91.116.7:4010 , piece_id=5G3O552GD5XO7SNIHST623QJBWJF2C7B6EN7ISGHHY3FYV6QL2DQ, seg_pos={0 0}, upload=18998]: Uploading piece... Aug 9 11:37:26.733124951 (18h11m21.164205082s) [upload=18998]: Next returned nil segment Aug 9 11:37:26.733142194 (18h11m21.164222325s) [upload=18998]: Waiting for error group managing segments... Aug 9 11:37:27.100757697 (18h11m21.531837848s) [noise=true , address=104.36.82.14:4222 , piece_id=Z4X6RJ5HBAYGUNKT6DXWSS2TEG2OUFPMH3NWYGA7D26XIFH46UAA, seg_pos={0 0}, upload=18998]: Done uploading piece. err: <nil> Aug 9 11:37:27.106781913 (18h11m21.537862064s) [noise=true , address=104.36.82.7:4912 , piece_id=MRA5X6N6MDQ2TADHZNIB6LBP5VQHFEGSYM3YNNZWNM2N4UU7JTCQ, seg_pos={0 0}, upload=18998]: Done uploading piece. err: <nil> Aug 9 11:37:27.136679105 (18h11m21.567759236s) [noise=true , address=104.36.82.5:4838 , piece_id=T3WCWCLB4WAXQY3S6VCFHLKYK6IY2Q6P2776CLGYS4QOFTXMYVEQ, seg_pos={0 0}, upload=18998]: Done uploading piece. err: <nil> Aug 9 11:37:27.139126447 (18h11m21.570206578s) [noise=true , address=104.36.82.3:4765 , piece_id=HBLTTPZYK5ZXNG2XV53QVJA3NPX3YWKEAAAFVMDXUO6BLR3QKCWQ, seg_pos={0 0}, upload=18998]: Done uploading piece. err: <nil> Aug 9 11:37:27.183514474 (18h11m21.614594606s) [noise=true , address=104.36.82.7:4909 , piece_id=OGG5ZCXSLIM5YKMM7K6KLKLWGOTX376DUTDT3567OB3AJIBPBKDQ, seg_pos={0 0}, upload=18998]: Done uploading piece. err: <nil> Aug 9 11:37:53.489259032 (18h11m47.920339163s) [noise=true , address=45.91.116.13:4155 , piece_id=K4CGD4OCR2B6ZWAZHB5MDBECVJB3AC2HDXS2TUVAVGIWQYJEARLA, seg_pos={0 0}, upload=18998]: Done uploading piece. err: <nil> Aug 9 11:37:54.124525138 (18h11m48.555605259s) [noise=true , address=45.91.116.7:4010 , piece_id=5G3O552GD5XO7SNIHST623QJBWJF2C7B6EN7ISGHHY3FYV6QL2DQ, seg_pos={0 0}, upload=18998]: Done uploading piece. err: ecclient: upload failed (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w, address:45.91.116.7:4010): protocol: write tcp 188.166.225.170:54960->45.91.116.7:4010: use of closed network connection; write tcp 188.166.225.170:54960->45.91.116.7:4010: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54960->45.91.116.7:4010: use of closed network connection Aug 9 11:37:54.124581035 (18h11m48.555661166s) [noise=true , address=77.90.31.10:4520 , piece_id=YRBBWM54FGU3B74JQ7EOAO7VSGVFXAVT4SNZ6KN3IQFIHIEVV5HA, seg_pos={0 0}, upload=18998]: Done uploading piece. err: ecclient: upload failed (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp, address:77.90.31.10:4520): protocol: write tcp 188.166.225.170:39560->77.90.31.10:4520: use of closed network connection; write tcp 188.166.225.170:39560->77.90.31.10:4520: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39560->77.90.31.10:4520: use of closed network connection Aug 9 11:37:54.124615480 (18h11m48.555695652s) [noise=true , address=45.91.116.12:4225 , piece_id=VFK2SMVTBT76D7YLQWGRA6TF47JD3B7J7F5NFR4E2MQVNGNHYTAQ, seg_pos={0 0}, upload=18998]: Done uploading piece. err: ecclient: upload failed (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy, address:45.91.116.12:4225): protocol: write tcp 188.166.225.170:38054->45.91.116.12:4225: use of closed network connection; write tcp 188.166.225.170:38054->45.91.116.12:4225: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38054->45.91.116.12:4225: use of closed network connection Aug 9 11:37:54.124643875 (18h11m48.555724016s) [noise=true , address=45.91.116.4:4940 , piece_id=M33ZGUCEF23HDGQZE5F5KLWQ43NHC7E7LKOVVGQ6P6TM2HFGB4WQ, seg_pos={0 0}, upload=18998]: Done uploading piece. err: ecclient: upload failed (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe, address:45.91.116.4:4940): protocol: write tcp 188.166.225.170:40630->45.91.116.4:4940: use of closed network connection; write tcp 188.166.225.170:40630->45.91.116.4:4940: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40630->45.91.116.4:4940: use of closed network connection Aug 9 11:37:54.124693159 (18h11m48.55577335s) [noise=true , address=45.91.116.14:4189 , piece_id=XVQMTYHB6H3IKSERYT2BEB4NP42KV6FRE6J2MKKU5WEOCVBHYIWA, seg_pos={0 0}, upload=18998]: Done uploading piece. err: ecclient: upload failed (node:1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N, address:45.91.116.14:4189): protocol: write tcp 188.166.225.170:37650->45.91.116.14:4189: use of closed network connection; write tcp 188.166.225.170:37650->45.91.116.14:4189: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37650->45.91.116.14:4189: use of closed network connection Aug 9 11:37:54.124771740 (18h11m48.555851871s) [noise=true , address=77.90.31.18:4806 , piece_id=QCURQYEJL22LBT2YFBR62RIHH7VNSGL7GERP66GKHXK3JP5WLVZQ, seg_pos={0 0}, upload=18998]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: write tcp 188.166.225.170:55536->77.90.31.18:4806: use of closed network connection; write tcp 188.166.225.170:55536->77.90.31.18:4806: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55536->77.90.31.18:4806: use of closed network connection Aug 9 11:37:54.124817217 (18h11m48.555897348s) [noise=true , address=77.90.31.15:4697 , piece_id=Q26SRGZO2YBJWPDIG4U422EICE6XEXPLHFHXLKFUBKMMJAZ4W7UQ, seg_pos={0 0}, upload=18998]: Done uploading piece. err: ecclient: upload failed (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, address:77.90.31.15:4697): protocol: write tcp 188.166.225.170:49618->77.90.31.15:4697: use of closed network connection; write tcp 188.166.225.170:49618->77.90.31.15:4697: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49618->77.90.31.15:4697: use of closed network connection Aug 9 11:37:54.124833387 (18h11m48.555913559s) [noise=true , address=45.91.116.6:4973 , piece_id=MHDNXTVKFMLRKGS7JIPL4U5VQMBXGP4EYM7JEGPSK4U3GIJ5JEHQ, seg_pos={0 0}, upload=18998]: Done uploading piece. err: ecclient: upload failed (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, address:45.91.116.6:4973): protocol: write tcp 188.166.225.170:59438->45.91.116.6:4973: use of closed network connection; write tcp 188.166.225.170:59438->45.91.116.6:4973: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59438->45.91.116.6:4973: use of closed network connection Aug 9 11:37:54.124840752 (18h11m48.555920883s) [noise=true , address=77.90.31.17:4772 , piece_id=MGZPHOODRIOTGABQJHNIP5FGF4XN6A723ZHUBKI6KNAJDKFWWOJA, seg_pos={0 0}, upload=18998]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: write tcp 188.166.225.170:42704->77.90.31.17:4772: use of closed network connection; write tcp 188.166.225.170:42704->77.90.31.17:4772: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42704->77.90.31.17:4772: use of closed network connection Aug 9 11:37:54.124845251 (18h11m48.555925382s) [noise=true , address=45.91.116.8:4047 , piece_id=B27B2QKZXICP4DHTDTCYSZZ7RN2PPCMKSRQBKRK5JJ4BE4RNWQAQ, seg_pos={0 0}, upload=18998]: Done uploading piece. err: ecclient: upload failed (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN, address:45.91.116.8:4047): protocol: write tcp 188.166.225.170:40794->45.91.116.8:4047: use of closed network connection; write tcp 188.166.225.170:40794->45.91.116.8:4047: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40794->45.91.116.8:4047: use of closed network connection Aug 9 11:37:54.124866321 (18h11m48.555946492s) [seg_pos={0 0}, upload=18998]: Done waiting for segment. successful: 6 optimal: 13 errs: <nil> Aug 9 11:37:54.353990811 (18h11m48.785070942s) [upload=18998]: Done uploading segments... Aug 9 11:37:54.354028673 (18h11m48.785108804s) [upload=18998]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 6) Duration: 5.786294803s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 3 out of 60 is failed (threshold=3) [.......................................X............X.X.....] duration: 5.638970609s FAIL (2025-08-09 12:03:39 UTC) orbiter warning: operation: ListObjects ======ERROR====== unable to find key in bucket: sj://uptime-checks-placement-20/orbiter_us_0/4M ======LOGS====== Aug 9 12:00:23.108483377 (18h34m17.539563508s) [upload=19402]: Starting upload Aug 9 12:00:23.108636078 (18h34m17.53971621s) [upload=19402]: Done starting upload Aug 9 12:00:23.108746840 (18h34m17.539826981s) [upload=19402]: Uploading segments... Aug 9 12:00:23.108904442 (18h34m17.539984573s) [upload=19402]: Got next segment. Inline: false Aug 9 12:00:23.108970418 (18h34m17.540050599s) [upload=19402]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 9 12:00:23.108977743 (18h34m17.540057904s) [upload=19402]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 9 12:00:23.579075368 (18h34m18.010155499s) [seg_pos={0 0}, upload=19402]: Begin upload segment... Aug 9 12:00:23.579127947 (18h34m18.010208078s) [seg_pos={0 0}, upload=19402]: Done begin upload segment. Aug 9 12:00:23.579157074 (18h34m18.010237205s) [noise=true , address=104.36.82.44:4899 , piece_id=KIQ5AFZ2FIUJ3XIG5ZQBJYITK5CD4ZDW5SYOUPFIYAOZBPENWFFA, seg_pos={0 0}, upload=19402]: Uploading piece... Aug 9 12:00:23.579206939 (18h34m18.01028707s) [noise=true , address=104.36.82.55:4199 , piece_id=6D4WT6KDBSMMCPXONV74YG4IXFEXPXU2FQTCL2W2NPZ2ERVMBUMQ, seg_pos={0 0}, upload=19402]: Uploading piece... Aug 9 12:00:23.579552951 (18h34m18.010633082s) [noise=true , address=64.49.13.139:4469 , piece_id=UTZT7WZ3LQ6LOZ3C6IXV5M7L3APATJOYDRZAMWUEYCZP4I34THCA, seg_pos={0 0}, upload=19402]: Uploading piece... Aug 9 12:00:23.579921666 (18h34m18.011001797s) [noise=true , address=104.36.82.59:4327 , piece_id=WJJFOV44IRLH2E6BFC46ALBJXD54ML4HQOFSUDFNJ3OJUXASSTZQ, seg_pos={0 0}, upload=19402]: Uploading piece... Aug 9 12:00:23.580275103 (18h34m18.011355234s) [noise=true , address=104.36.82.13:4197 , piece_id=NEMRRYQPHXM73GSZVEETU7UNMUDQVNKKSLKQTRSBHBKPQ35KLLPQ, seg_pos={0 0}, upload=19402]: Uploading piece... Aug 9 12:00:23.580699144 (18h34m18.011779275s) [noise=true , address=104.36.82.23:4454 , piece_id=VNKC653W6NNZTTH6CRDK2LSD5XFLLGJGYOOIELABDCKC364DITDA, seg_pos={0 0}, upload=19402]: Uploading piece... Aug 9 12:00:23.580803934 (18h34m18.011884065s) [noise=true , address=104.36.82.61:4396 , piece_id=LJ73GMLI4ZDNLDSGIRT4GRBU56YFUOBTOC7OW3NS4TRWSUXMQCHA, seg_pos={0 0}, upload=19402]: Uploading piece... Aug 9 12:00:23.581071286 (18h34m18.012151417s) [noise=true , address=104.36.82.45:4934 , piece_id=K7XWI37JOPKYII6BFSO6AZT73S3FYOH5JKCJXOM5O5SCWAG62OIQ, seg_pos={0 0}, upload=19402]: Uploading piece... Aug 9 12:00:23.581305595 (18h34m18.012385726s) [noise=true , address=104.36.82.60:4354 , piece_id=HR52C2IWZNCL2XI553EUKTS37R5N4ILVW34GJSPWJV3Z3W7KKGWA, seg_pos={0 0}, upload=19402]: Uploading piece... Aug 9 12:00:23.581541305 (18h34m18.012621436s) [noise=true , address=45.91.116.12:4243 , piece_id=FZW53SVI3PFJ3M6WQE74FPB737JEBGBGA2YB53XCU5DEPEUQHU7Q, seg_pos={0 0}, upload=19402]: Uploading piece... Aug 9 12:00:23.581620086 (18h34m18.012700217s) [noise=true , address=104.36.82.58:4268 , piece_id=QLMKG6PZ4XYJJUTOHFW7EWTFGTST5HYLTHP5MMX6X3AJJJAZSZ3Q, seg_pos={0 0}, upload=19402]: Uploading piece... Aug 9 12:00:23.581922384 (18h34m18.013002515s) [noise=true , address=104.36.82.9:4031 , piece_id=LLSOV26QMVJRDNGWSOKPT4MIGAMTHU6AIADGGJNG6526XAR3SF6Q, seg_pos={0 0}, upload=19402]: Uploading piece... Aug 9 12:00:23.582143177 (18h34m18.013223308s) [noise=true , address=104.36.82.60:4358 , piece_id=WPW7FORNXWY6DOAXVH775D4W3VFOH7PRMQO5UXKJNIFFCRVSE43A, seg_pos={0 0}, upload=19402]: Uploading piece... Aug 9 12:00:23.582347538 (18h34m18.013427669s) [noise=true , address=104.36.82.14:4242 , piece_id=2JREAT5B4ZOSHBJPMPKVQOW2VDMLQIDURO7KTYRPFJAM6U5K57WQ, seg_pos={0 0}, upload=19402]: Uploading piece... Aug 9 12:00:23.582596004 (18h34m18.013676135s) [noise=true , address=64.49.13.170:4726 , piece_id=A2OWN4ZBVO3MZKLRAHERFMYAZDSLB4HMYESALCEVEH66LGUZEQOQ, seg_pos={0 0}, upload=19402]: Uploading piece... Aug 9 12:00:23.582774395 (18h34m18.013854526s) [noise=true , address=104.36.82.46:4484 , piece_id=FQHBXDSLXK2GIOSY4V5QBBPLOH7ZD3KHJVXR5IFYPMICQCWH7UFA, seg_pos={0 0}, upload=19402]: Uploading piece... Aug 9 12:00:23.583063899 (18h34m18.01414403s) [noise=true , address=77.90.31.12:4604 , piece_id=HPMFA765VZOJGNXPWE5SCA4AAK7QB4HFGUSYFQWOFLVZTU6PSUMA, seg_pos={0 0}, upload=19402]: Uploading piece... Aug 9 12:00:23.583138481 (18h34m18.014218612s) [noise=true , address=104.36.82.5:4845 , piece_id=5AJNZJD6MFKE3U6EZ27IEXQEO4KOUGSXA7EBCDSWVTVFYD4AJTHQ, seg_pos={0 0}, upload=19402]: Uploading piece... Aug 9 12:00:24.683477824 (18h34m19.114557955s) [upload=19402]: Next returned nil segment Aug 9 12:00:24.683489506 (18h34m19.114569637s) [upload=19402]: Waiting for error group managing segments... Aug 9 12:00:24.955322603 (18h34m19.386402734s) [noise=true , address=64.49.13.170:4726 , piece_id=A2OWN4ZBVO3MZKLRAHERFMYAZDSLB4HMYESALCEVEH66LGUZEQOQ, seg_pos={0 0}, upload=19402]: Done uploading piece. err: <nil> Aug 9 12:00:24.969832404 (18h34m19.400912535s) [noise=true , address=64.49.13.139:4469 , piece_id=UTZT7WZ3LQ6LOZ3C6IXV5M7L3APATJOYDRZAMWUEYCZP4I34THCA, seg_pos={0 0}, upload=19402]: Done uploading piece. err: <nil> Aug 9 12:00:25.516280300 (18h34m19.947360431s) [noise=true , address=104.36.82.55:4199 , piece_id=6D4WT6KDBSMMCPXONV74YG4IXFEXPXU2FQTCL2W2NPZ2ERVMBUMQ, seg_pos={0 0}, upload=19402]: Done uploading piece. err: <nil> Aug 9 12:00:25.525386162 (18h34m19.956466293s) [noise=true , address=104.36.82.46:4484 , piece_id=FQHBXDSLXK2GIOSY4V5QBBPLOH7ZD3KHJVXR5IFYPMICQCWH7UFA, seg_pos={0 0}, upload=19402]: Done uploading piece. err: <nil> Aug 9 12:00:25.526709204 (18h34m19.957789335s) [noise=true , address=104.36.82.60:4358 , piece_id=WPW7FORNXWY6DOAXVH775D4W3VFOH7PRMQO5UXKJNIFFCRVSE43A, seg_pos={0 0}, upload=19402]: Done uploading piece. err: <nil> Aug 9 12:00:25.530790893 (18h34m19.961871024s) [noise=true , address=104.36.82.44:4899 , piece_id=KIQ5AFZ2FIUJ3XIG5ZQBJYITK5CD4ZDW5SYOUPFIYAOZBPENWFFA, seg_pos={0 0}, upload=19402]: Done uploading piece. err: <nil> Aug 9 12:00:25.531900687 (18h34m19.962980818s) [noise=true , address=104.36.82.58:4268 , piece_id=QLMKG6PZ4XYJJUTOHFW7EWTFGTST5HYLTHP5MMX6X3AJJJAZSZ3Q, seg_pos={0 0}, upload=19402]: Done uploading piece. err: <nil> Aug 9 12:00:25.536797967 (18h34m19.967878098s) [noise=true , address=104.36.82.9:4031 , piece_id=LLSOV26QMVJRDNGWSOKPT4MIGAMTHU6AIADGGJNG6526XAR3SF6Q, seg_pos={0 0}, upload=19402]: Done uploading piece. err: <nil> Aug 9 12:00:25.545685641 (18h34m19.976765772s) [noise=true , address=104.36.82.60:4354 , piece_id=HR52C2IWZNCL2XI553EUKTS37R5N4ILVW34GJSPWJV3Z3W7KKGWA, seg_pos={0 0}, upload=19402]: Done uploading piece. err: <nil> Aug 9 12:00:25.555951362 (18h34m19.987031493s) [noise=true , address=104.36.82.61:4396 , piece_id=LJ73GMLI4ZDNLDSGIRT4GRBU56YFUOBTOC7OW3NS4TRWSUXMQCHA, seg_pos={0 0}, upload=19402]: Done uploading piece. err: <nil> Aug 9 12:00:25.566816712 (18h34m19.997896843s) [noise=true , address=104.36.82.5:4845 , piece_id=5AJNZJD6MFKE3U6EZ27IEXQEO4KOUGSXA7EBCDSWVTVFYD4AJTHQ, seg_pos={0 0}, upload=19402]: Done uploading piece. err: <nil> Aug 9 12:00:25.585861129 (18h34m20.01694126s) [noise=true , address=104.36.82.45:4934 , piece_id=K7XWI37JOPKYII6BFSO6AZT73S3FYOH5JKCJXOM5O5SCWAG62OIQ, seg_pos={0 0}, upload=19402]: Done uploading piece. err: <nil> Aug 9 12:00:25.594196727 (18h34m20.025276858s) [noise=true , address=45.91.116.12:4243 , piece_id=FZW53SVI3PFJ3M6WQE74FPB737JEBGBGA2YB53XCU5DEPEUQHU7Q, seg_pos={0 0}, upload=19402]: Done uploading piece. err: <nil> Aug 9 12:00:25.606123457 (18h34m20.037203588s) [noise=true , address=77.90.31.12:4604 , piece_id=HPMFA765VZOJGNXPWE5SCA4AAK7QB4HFGUSYFQWOFLVZTU6PSUMA, seg_pos={0 0}, upload=19402]: Done uploading piece. err: <nil> Aug 9 12:00:25.606140419 (18h34m20.03722055s) [seg_pos={0 0}, upload=19402]: Segment reached optimal threshold of 14 pieces. Aug 9 12:00:25.606241462 (18h34m20.037321593s) [noise=true , address=104.36.82.13:4197 , piece_id=NEMRRYQPHXM73GSZVEETU7UNMUDQVNKKSLKQTRSBHBKPQ35KLLPQ, seg_pos={0 0}, upload=19402]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1cB89C5xnya9WGD9dGBsSouqkZH3nLqytrg9Qo4UH2kdFonTjM): protocol: expected piece hash; context canceled; EOF Aug 9 12:00:25.606346553 (18h34m20.037426675s) [noise=true , address=104.36.82.14:4242 , piece_id=2JREAT5B4ZOSHBJPMPKVQOW2VDMLQIDURO7KTYRPFJAM6U5K57WQ, seg_pos={0 0}, upload=19402]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1inR8rjUp4fRQWcaD2S7MRxveHyFxKoutWoEiuKQG3C895PaMz): protocol: expected piece hash; context canceled; EOF Aug 9 12:00:25.606405596 (18h34m20.037485727s) [noise=true , address=104.36.82.23:4454 , piece_id=VNKC653W6NNZTTH6CRDK2LSD5XFLLGJGYOOIELABDCKC364DITDA, seg_pos={0 0}, upload=19402]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1UCkuYGdVYLoR8oc1jpCZCN2smNfVqFwC8a7q95fJb8haB3MY2): protocol: expected piece hash; context canceled; EOF Aug 9 12:00:25.606463838 (18h34m20.037543969s) [noise=true , address=104.36.82.59:4327 , piece_id=WJJFOV44IRLH2E6BFC46ALBJXD54ML4HQOFSUDFNJ3OJUXASSTZQ, seg_pos={0 0}, upload=19402]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1V4C6AWGbyYQWfinoSj7iKovHeTJ9hqypyC5U9rZ8pveXBBfCY): protocol: expected piece hash; context canceled; EOF Aug 9 12:00:25.606478186 (18h34m20.037558317s) [seg_pos={0 0}, upload=19402]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 9 12:00:25.606535966 (18h34m20.037616097s) [upload=19402]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 9 12:00:25.606540855 (18h34m20.037620976s) [upload=19402]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 9 12:00:25.913625167 (18h34m20.344705298s) [upload=19402]: Done uploading segments... Aug 9 12:00:25.913644565 (18h34m20.344724696s) [upload=19402]: Upload finished. err: <nil> Duration: 5.638995476s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 3 out of 60 is failed (threshold=3) [........................................X............X.X....] duration: 5.838065276s FAIL (2025-08-09 12:03:20 UTC) orbiter warning: operation: ListObjects ======ERROR====== unable to find key in bucket: sj://uptime-checks-placement-20/orbiter_us_0/4M ======LOGS====== Aug 9 12:00:23.108483377 (18h34m17.539563508s) [upload=19402]: Starting upload Aug 9 12:00:23.108636078 (18h34m17.53971621s) [upload=19402]: Done starting upload Aug 9 12:00:23.108746840 (18h34m17.539826981s) [upload=19402]: Uploading segments... Aug 9 12:00:23.108904442 (18h34m17.539984573s) [upload=19402]: Got next segment. Inline: false Aug 9 12:00:23.108970418 (18h34m17.540050599s) [upload=19402]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 9 12:00:23.108977743 (18h34m17.540057904s) [upload=19402]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 9 12:00:23.579075368 (18h34m18.010155499s) [seg_pos={0 0}, upload=19402]: Begin upload segment... Aug 9 12:00:23.579127947 (18h34m18.010208078s) [seg_pos={0 0}, upload=19402]: Done begin upload segment. Aug 9 12:00:23.579157074 (18h34m18.010237205s) [noise=true , address=104.36.82.44:4899 , piece_id=KIQ5AFZ2FIUJ3XIG5ZQBJYITK5CD4ZDW5SYOUPFIYAOZBPENWFFA, seg_pos={0 0}, upload=19402]: Uploading piece... Aug 9 12:00:23.579206939 (18h34m18.01028707s) [noise=true , address=104.36.82.55:4199 , piece_id=6D4WT6KDBSMMCPXONV74YG4IXFEXPXU2FQTCL2W2NPZ2ERVMBUMQ, seg_pos={0 0}, upload=19402]: Uploading piece... Aug 9 12:00:23.579552951 (18h34m18.010633082s) [noise=true , address=64.49.13.139:4469 , piece_id=UTZT7WZ3LQ6LOZ3C6IXV5M7L3APATJOYDRZAMWUEYCZP4I34THCA, seg_pos={0 0}, upload=19402]: Uploading piece... Aug 9 12:00:23.579921666 (18h34m18.011001797s) [noise=true , address=104.36.82.59:4327 , piece_id=WJJFOV44IRLH2E6BFC46ALBJXD54ML4HQOFSUDFNJ3OJUXASSTZQ, seg_pos={0 0}, upload=19402]: Uploading piece... Aug 9 12:00:23.580275103 (18h34m18.011355234s) [noise=true , address=104.36.82.13:4197 , piece_id=NEMRRYQPHXM73GSZVEETU7UNMUDQVNKKSLKQTRSBHBKPQ35KLLPQ, seg_pos={0 0}, upload=19402]: Uploading piece... Aug 9 12:00:23.580699144 (18h34m18.011779275s) [noise=true , address=104.36.82.23:4454 , piece_id=VNKC653W6NNZTTH6CRDK2LSD5XFLLGJGYOOIELABDCKC364DITDA, seg_pos={0 0}, upload=19402]: Uploading piece... Aug 9 12:00:23.580803934 (18h34m18.011884065s) [noise=true , address=104.36.82.61:4396 , piece_id=LJ73GMLI4ZDNLDSGIRT4GRBU56YFUOBTOC7OW3NS4TRWSUXMQCHA, seg_pos={0 0}, upload=19402]: Uploading piece... Aug 9 12:00:23.581071286 (18h34m18.012151417s) [noise=true , address=104.36.82.45:4934 , piece_id=K7XWI37JOPKYII6BFSO6AZT73S3FYOH5JKCJXOM5O5SCWAG62OIQ, seg_pos={0 0}, upload=19402]: Uploading piece... Aug 9 12:00:23.581305595 (18h34m18.012385726s) [noise=true , address=104.36.82.60:4354 , piece_id=HR52C2IWZNCL2XI553EUKTS37R5N4ILVW34GJSPWJV3Z3W7KKGWA, seg_pos={0 0}, upload=19402]: Uploading piece... Aug 9 12:00:23.581541305 (18h34m18.012621436s) [noise=true , address=45.91.116.12:4243 , piece_id=FZW53SVI3PFJ3M6WQE74FPB737JEBGBGA2YB53XCU5DEPEUQHU7Q, seg_pos={0 0}, upload=19402]: Uploading piece... Aug 9 12:00:23.581620086 (18h34m18.012700217s) [noise=true , address=104.36.82.58:4268 , piece_id=QLMKG6PZ4XYJJUTOHFW7EWTFGTST5HYLTHP5MMX6X3AJJJAZSZ3Q, seg_pos={0 0}, upload=19402]: Uploading piece... Aug 9 12:00:23.581922384 (18h34m18.013002515s) [noise=true , address=104.36.82.9:4031 , piece_id=LLSOV26QMVJRDNGWSOKPT4MIGAMTHU6AIADGGJNG6526XAR3SF6Q, seg_pos={0 0}, upload=19402]: Uploading piece... Aug 9 12:00:23.582143177 (18h34m18.013223308s) [noise=true , address=104.36.82.60:4358 , piece_id=WPW7FORNXWY6DOAXVH775D4W3VFOH7PRMQO5UXKJNIFFCRVSE43A, seg_pos={0 0}, upload=19402]: Uploading piece... Aug 9 12:00:23.582347538 (18h34m18.013427669s) [noise=true , address=104.36.82.14:4242 , piece_id=2JREAT5B4ZOSHBJPMPKVQOW2VDMLQIDURO7KTYRPFJAM6U5K57WQ, seg_pos={0 0}, upload=19402]: Uploading piece... Aug 9 12:00:23.582596004 (18h34m18.013676135s) [noise=true , address=64.49.13.170:4726 , piece_id=A2OWN4ZBVO3MZKLRAHERFMYAZDSLB4HMYESALCEVEH66LGUZEQOQ, seg_pos={0 0}, upload=19402]: Uploading piece... Aug 9 12:00:23.582774395 (18h34m18.013854526s) [noise=true , address=104.36.82.46:4484 , piece_id=FQHBXDSLXK2GIOSY4V5QBBPLOH7ZD3KHJVXR5IFYPMICQCWH7UFA, seg_pos={0 0}, upload=19402]: Uploading piece... Aug 9 12:00:23.583063899 (18h34m18.01414403s) [noise=true , address=77.90.31.12:4604 , piece_id=HPMFA765VZOJGNXPWE5SCA4AAK7QB4HFGUSYFQWOFLVZTU6PSUMA, seg_pos={0 0}, upload=19402]: Uploading piece... Aug 9 12:00:23.583138481 (18h34m18.014218612s) [noise=true , address=104.36.82.5:4845 , piece_id=5AJNZJD6MFKE3U6EZ27IEXQEO4KOUGSXA7EBCDSWVTVFYD4AJTHQ, seg_pos={0 0}, upload=19402]: Uploading piece... Aug 9 12:00:24.683477824 (18h34m19.114557955s) [upload=19402]: Next returned nil segment Aug 9 12:00:24.683489506 (18h34m19.114569637s) [upload=19402]: Waiting for error group managing segments... Aug 9 12:00:24.955322603 (18h34m19.386402734s) [noise=true , address=64.49.13.170:4726 , piece_id=A2OWN4ZBVO3MZKLRAHERFMYAZDSLB4HMYESALCEVEH66LGUZEQOQ, seg_pos={0 0}, upload=19402]: Done uploading piece. err: <nil> Aug 9 12:00:24.969832404 (18h34m19.400912535s) [noise=true , address=64.49.13.139:4469 , piece_id=UTZT7WZ3LQ6LOZ3C6IXV5M7L3APATJOYDRZAMWUEYCZP4I34THCA, seg_pos={0 0}, upload=19402]: Done uploading piece. err: <nil> Aug 9 12:00:25.516280300 (18h34m19.947360431s) [noise=true , address=104.36.82.55:4199 , piece_id=6D4WT6KDBSMMCPXONV74YG4IXFEXPXU2FQTCL2W2NPZ2ERVMBUMQ, seg_pos={0 0}, upload=19402]: Done uploading piece. err: <nil> Aug 9 12:00:25.525386162 (18h34m19.956466293s) [noise=true , address=104.36.82.46:4484 , piece_id=FQHBXDSLXK2GIOSY4V5QBBPLOH7ZD3KHJVXR5IFYPMICQCWH7UFA, seg_pos={0 0}, upload=19402]: Done uploading piece. err: <nil> Aug 9 12:00:25.526709204 (18h34m19.957789335s) [noise=true , address=104.36.82.60:4358 , piece_id=WPW7FORNXWY6DOAXVH775D4W3VFOH7PRMQO5UXKJNIFFCRVSE43A, seg_pos={0 0}, upload=19402]: Done uploading piece. err: <nil> Aug 9 12:00:25.530790893 (18h34m19.961871024s) [noise=true , address=104.36.82.44:4899 , piece_id=KIQ5AFZ2FIUJ3XIG5ZQBJYITK5CD4ZDW5SYOUPFIYAOZBPENWFFA, seg_pos={0 0}, upload=19402]: Done uploading piece. err: <nil> Aug 9 12:00:25.531900687 (18h34m19.962980818s) [noise=true , address=104.36.82.58:4268 , piece_id=QLMKG6PZ4XYJJUTOHFW7EWTFGTST5HYLTHP5MMX6X3AJJJAZSZ3Q, seg_pos={0 0}, upload=19402]: Done uploading piece. err: <nil> Aug 9 12:00:25.536797967 (18h34m19.967878098s) [noise=true , address=104.36.82.9:4031 , piece_id=LLSOV26QMVJRDNGWSOKPT4MIGAMTHU6AIADGGJNG6526XAR3SF6Q, seg_pos={0 0}, upload=19402]: Done uploading piece. err: <nil> Aug 9 12:00:25.545685641 (18h34m19.976765772s) [noise=true , address=104.36.82.60:4354 , piece_id=HR52C2IWZNCL2XI553EUKTS37R5N4ILVW34GJSPWJV3Z3W7KKGWA, seg_pos={0 0}, upload=19402]: Done uploading piece. err: <nil> Aug 9 12:00:25.555951362 (18h34m19.987031493s) [noise=true , address=104.36.82.61:4396 , piece_id=LJ73GMLI4ZDNLDSGIRT4GRBU56YFUOBTOC7OW3NS4TRWSUXMQCHA, seg_pos={0 0}, upload=19402]: Done uploading piece. err: <nil> Aug 9 12:00:25.566816712 (18h34m19.997896843s) [noise=true , address=104.36.82.5:4845 , piece_id=5AJNZJD6MFKE3U6EZ27IEXQEO4KOUGSXA7EBCDSWVTVFYD4AJTHQ, seg_pos={0 0}, upload=19402]: Done uploading piece. err: <nil> Aug 9 12:00:25.585861129 (18h34m20.01694126s) [noise=true , address=104.36.82.45:4934 , piece_id=K7XWI37JOPKYII6BFSO6AZT73S3FYOH5JKCJXOM5O5SCWAG62OIQ, seg_pos={0 0}, upload=19402]: Done uploading piece. err: <nil> Aug 9 12:00:25.594196727 (18h34m20.025276858s) [noise=true , address=45.91.116.12:4243 , piece_id=FZW53SVI3PFJ3M6WQE74FPB737JEBGBGA2YB53XCU5DEPEUQHU7Q, seg_pos={0 0}, upload=19402]: Done uploading piece. err: <nil> Aug 9 12:00:25.606123457 (18h34m20.037203588s) [noise=true , address=77.90.31.12:4604 , piece_id=HPMFA765VZOJGNXPWE5SCA4AAK7QB4HFGUSYFQWOFLVZTU6PSUMA, seg_pos={0 0}, upload=19402]: Done uploading piece. err: <nil> Aug 9 12:00:25.606140419 (18h34m20.03722055s) [seg_pos={0 0}, upload=19402]: Segment reached optimal threshold of 14 pieces. Aug 9 12:00:25.606241462 (18h34m20.037321593s) [noise=true , address=104.36.82.13:4197 , piece_id=NEMRRYQPHXM73GSZVEETU7UNMUDQVNKKSLKQTRSBHBKPQ35KLLPQ, seg_pos={0 0}, upload=19402]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1cB89C5xnya9WGD9dGBsSouqkZH3nLqytrg9Qo4UH2kdFonTjM): protocol: expected piece hash; context canceled; EOF Aug 9 12:00:25.606346553 (18h34m20.037426675s) [noise=true , address=104.36.82.14:4242 , piece_id=2JREAT5B4ZOSHBJPMPKVQOW2VDMLQIDURO7KTYRPFJAM6U5K57WQ, seg_pos={0 0}, upload=19402]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1inR8rjUp4fRQWcaD2S7MRxveHyFxKoutWoEiuKQG3C895PaMz): protocol: expected piece hash; context canceled; EOF Aug 9 12:00:25.606405596 (18h34m20.037485727s) [noise=true , address=104.36.82.23:4454 , piece_id=VNKC653W6NNZTTH6CRDK2LSD5XFLLGJGYOOIELABDCKC364DITDA, seg_pos={0 0}, upload=19402]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1UCkuYGdVYLoR8oc1jpCZCN2smNfVqFwC8a7q95fJb8haB3MY2): protocol: expected piece hash; context canceled; EOF Aug 9 12:00:25.606463838 (18h34m20.037543969s) [noise=true , address=104.36.82.59:4327 , piece_id=WJJFOV44IRLH2E6BFC46ALBJXD54ML4HQOFSUDFNJ3OJUXASSTZQ, seg_pos={0 0}, upload=19402]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1V4C6AWGbyYQWfinoSj7iKovHeTJ9hqypyC5U9rZ8pveXBBfCY): protocol: expected piece hash; context canceled; EOF Aug 9 12:00:25.606478186 (18h34m20.037558317s) [seg_pos={0 0}, upload=19402]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 9 12:00:25.606535966 (18h34m20.037616097s) [upload=19402]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 9 12:00:25.606540855 (18h34m20.037620976s) [upload=19402]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 9 12:00:25.913625167 (18h34m20.344705298s) [upload=19402]: Done uploading segments... Aug 9 12:00:25.913644565 (18h34m20.344724696s) [upload=19402]: Upload finished. err: <nil> Duration: 5.838098749s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 3 out of 60 is failed (threshold=3) [.........................X..................X.........X.....] duration: 1.98647421s FAIL (2025-08-09 12:03:15 UTC) orbiter warning: operation: ListObjects ======ERROR====== unable to find key in bucket: sj://uptime-checks-placement-20/orbiter_us_0/1K ======LOGS====== Aug 9 11:59:31.819436107 (18h33m26.250516238s) [upload=19388]: Starting upload Aug 9 11:59:31.819492355 (18h33m26.250572516s) [upload=19388]: Done starting upload Aug 9 11:59:31.819563080 (18h33m26.250643211s) [upload=19388]: Uploading segments... Aug 9 11:59:31.819616853 (18h33m26.250696984s) [upload=19388]: Got next segment. Inline: true Aug 9 11:59:31.819621552 (18h33m26.250701683s) [upload=19388]: Waiting for error group managing segments... Aug 9 11:59:31.819660466 (18h33m26.250740597s) [upload=19388]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 9 11:59:31.819665206 (18h33m26.250745327s) [upload=19388]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 9 11:59:31.819667851 (18h33m26.250747972s) [upload=19388]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 9 11:59:32.242597809 (18h33m26.67367794s) [upload=19388]: Done uploading segments... Aug 9 11:59:32.242615142 (18h33m26.673695263s) [upload=19388]: Upload finished. err: <nil> Duration:* 1.986499998s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................X.................................] duration: 5.972478636s FAIL (2025-08-09 12:03:08 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 6) ======LOGS====== ADHZNIB6LBP5VQHFEGSYM3YNNZWNM2N4UU7JTCQ, seg_pos={0 0}, upload=18998]: Uploading piece... Aug 9 11:37:25.005421531 (18h11m19.436501662s) [noise=true , address=45.91.116.4:4940 , piece_id=M33ZGUCEF23HDGQZE5F5KLWQ43NHC7E7LKOVVGQ6P6TM2HFGB4WQ, seg_pos={0 0}, upload=18998]: Uploading piece... Aug 9 11:37:25.005614329 (18h11m19.43669446s) [noise=true , address=45.91.116.8:4047 , piece_id=B27B2QKZXICP4DHTDTCYSZZ7RN2PPCMKSRQBKRK5JJ4BE4RNWQAQ, seg_pos={0 0}, upload=18998]: Uploading piece... Aug 9 11:37:25.005727536 (18h11m19.436807667s) [noise=true , address=45.91.116.12:4225 , piece_id=VFK2SMVTBT76D7YLQWGRA6TF47JD3B7J7F5NFR4E2MQVNGNHYTAQ, seg_pos={0 0}, upload=18998]: Uploading piece... Aug 9 11:37:25.006281517 (18h11m19.437361648s) [noise=true , address=104.36.82.7:4909 , piece_id=OGG5ZCXSLIM5YKMM7K6KLKLWGOTX376DUTDT3567OB3AJIBPBKDQ, seg_pos={0 0}, upload=18998]: Uploading piece... Aug 9 11:37:25.006632558 (18h11m19.437712679s) [noise=true , address=45.91.116.13:4155 , piece_id=K4CGD4OCR2B6ZWAZHB5MDBECVJB3AC2HDXS2TUVAVGIWQYJEARLA, seg_pos={0 0}, upload=18998]: Uploading piece... Aug 9 11:37:25.006996264 (18h11m19.438076395s) [noise=true , address=77.90.31.18:4806 , piece_id=QCURQYEJL22LBT2YFBR62RIHH7VNSGL7GERP66GKHXK3JP5WLVZQ, seg_pos={0 0}, upload=18998]: Uploading piece... Aug 9 11:37:25.007245231 (18h11m19.438325362s) [noise=true , address=104.36.82.3:4765 , piece_id=HBLTTPZYK5ZXNG2XV53QVJA3NPX3YWKEAAAFVMDXUO6BLR3QKCWQ, seg_pos={0 0}, upload=18998]: Uploading piece... Aug 9 11:37:25.007475481 (18h11m19.438555612s) [noise=true , address=45.91.116.6:4973 , piece_id=MHDNXTVKFMLRKGS7JIPL4U5VQMBXGP4EYM7JEGPSK4U3GIJ5JEHQ, seg_pos={0 0}, upload=18998]: Uploading piece... Aug 9 11:37:25.007873984 (18h11m19.438954115s) [noise=true , address=45.91.116.7:4010 , piece_id=5G3O552GD5XO7SNIHST623QJBWJF2C7B6EN7ISGHHY3FYV6QL2DQ, seg_pos={0 0}, upload=18998]: Uploading piece... Aug 9 11:37:26.733124951 (18h11m21.164205082s) [upload=18998]: Next returned nil segment Aug 9 11:37:26.733142194 (18h11m21.164222325s) [upload=18998]: Waiting for error group managing segments... Aug 9 11:37:27.100757697 (18h11m21.531837848s) [noise=true , address=104.36.82.14:4222 , piece_id=Z4X6RJ5HBAYGUNKT6DXWSS2TEG2OUFPMH3NWYGA7D26XIFH46UAA, seg_pos={0 0}, upload=18998]: Done uploading piece. err: <nil> Aug 9 11:37:27.106781913 (18h11m21.537862064s) [noise=true , address=104.36.82.7:4912 , piece_id=MRA5X6N6MDQ2TADHZNIB6LBP5VQHFEGSYM3YNNZWNM2N4UU7JTCQ, seg_pos={0 0}, upload=18998]: Done uploading piece. err: <nil> Aug 9 11:37:27.136679105 (18h11m21.567759236s) [noise=true , address=104.36.82.5:4838 , piece_id=T3WCWCLB4WAXQY3S6VCFHLKYK6IY2Q6P2776CLGYS4QOFTXMYVEQ, seg_pos={0 0}, upload=18998]: Done uploading piece. err: <nil> Aug 9 11:37:27.139126447 (18h11m21.570206578s) [noise=true , address=104.36.82.3:4765 , piece_id=HBLTTPZYK5ZXNG2XV53QVJA3NPX3YWKEAAAFVMDXUO6BLR3QKCWQ, seg_pos={0 0}, upload=18998]: Done uploading piece. err: <nil> Aug 9 11:37:27.183514474 (18h11m21.614594606s) [noise=true , address=104.36.82.7:4909 , piece_id=OGG5ZCXSLIM5YKMM7K6KLKLWGOTX376DUTDT3567OB3AJIBPBKDQ, seg_pos={0 0}, upload=18998]: Done uploading piece. err: <nil> Aug 9 11:37:53.489259032 (18h11m47.920339163s) [noise=true , address=45.91.116.13:4155 , piece_id=K4CGD4OCR2B6ZWAZHB5MDBECVJB3AC2HDXS2TUVAVGIWQYJEARLA, seg_pos={0 0}, upload=18998]: Done uploading piece. err: <nil> Aug 9 11:37:54.124525138 (18h11m48.555605259s) [noise=true , address=45.91.116.7:4010 , piece_id=5G3O552GD5XO7SNIHST623QJBWJF2C7B6EN7ISGHHY3FYV6QL2DQ, seg_pos={0 0}, upload=18998]: Done uploading piece. err: ecclient: upload failed (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w, address:45.91.116.7:4010): protocol: write tcp 188.166.225.170:54960->45.91.116.7:4010: use of closed network connection; write tcp 188.166.225.170:54960->45.91.116.7:4010: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54960->45.91.116.7:4010: use of closed network connection Aug 9 11:37:54.124581035 (18h11m48.555661166s) [noise=true , address=77.90.31.10:4520 , piece_id=YRBBWM54FGU3B74JQ7EOAO7VSGVFXAVT4SNZ6KN3IQFIHIEVV5HA, seg_pos={0 0}, upload=18998]: Done uploading piece. err: ecclient: upload failed (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp, address:77.90.31.10:4520): protocol: write tcp 188.166.225.170:39560->77.90.31.10:4520: use of closed network connection; write tcp 188.166.225.170:39560->77.90.31.10:4520: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39560->77.90.31.10:4520: use of closed network connection Aug 9 11:37:54.124615480 (18h11m48.555695652s) [noise=true , address=45.91.116.12:4225 , piece_id=VFK2SMVTBT76D7YLQWGRA6TF47JD3B7J7F5NFR4E2MQVNGNHYTAQ, seg_pos={0 0}, upload=18998]: Done uploading piece. err: ecclient: upload failed (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy, address:45.91.116.12:4225): protocol: write tcp 188.166.225.170:38054->45.91.116.12:4225: use of closed network connection; write tcp 188.166.225.170:38054->45.91.116.12:4225: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38054->45.91.116.12:4225: use of closed network connection Aug 9 11:37:54.124643875 (18h11m48.555724016s) [noise=true , address=45.91.116.4:4940 , piece_id=M33ZGUCEF23HDGQZE5F5KLWQ43NHC7E7LKOVVGQ6P6TM2HFGB4WQ, seg_pos={0 0}, upload=18998]: Done uploading piece. err: ecclient: upload failed (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe, address:45.91.116.4:4940): protocol: write tcp 188.166.225.170:40630->45.91.116.4:4940: use of closed network connection; write tcp 188.166.225.170:40630->45.91.116.4:4940: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40630->45.91.116.4:4940: use of closed network connection Aug 9 11:37:54.124693159 (18h11m48.55577335s) [noise=true , address=45.91.116.14:4189 , piece_id=XVQMTYHB6H3IKSERYT2BEB4NP42KV6FRE6J2MKKU5WEOCVBHYIWA, seg_pos={0 0}, upload=18998]: Done uploading piece. err: ecclient: upload failed (node:1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N, address:45.91.116.14:4189): protocol: write tcp 188.166.225.170:37650->45.91.116.14:4189: use of closed network connection; write tcp 188.166.225.170:37650->45.91.116.14:4189: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37650->45.91.116.14:4189: use of closed network connection Aug 9 11:37:54.124771740 (18h11m48.555851871s) [noise=true , address=77.90.31.18:4806 , piece_id=QCURQYEJL22LBT2YFBR62RIHH7VNSGL7GERP66GKHXK3JP5WLVZQ, seg_pos={0 0}, upload=18998]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: write tcp 188.166.225.170:55536->77.90.31.18:4806: use of closed network connection; write tcp 188.166.225.170:55536->77.90.31.18:4806: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55536->77.90.31.18:4806: use of closed network connection Aug 9 11:37:54.124817217 (18h11m48.555897348s) [noise=true , address=77.90.31.15:4697 , piece_id=Q26SRGZO2YBJWPDIG4U422EICE6XEXPLHFHXLKFUBKMMJAZ4W7UQ, seg_pos={0 0}, upload=18998]: Done uploading piece. err: ecclient: upload failed (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, address:77.90.31.15:4697): protocol: write tcp 188.166.225.170:49618->77.90.31.15:4697: use of closed network connection; write tcp 188.166.225.170:49618->77.90.31.15:4697: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49618->77.90.31.15:4697: use of closed network connection Aug 9 11:37:54.124833387 (18h11m48.555913559s) [noise=true , address=45.91.116.6:4973 , piece_id=MHDNXTVKFMLRKGS7JIPL4U5VQMBXGP4EYM7JEGPSK4U3GIJ5JEHQ, seg_pos={0 0}, upload=18998]: Done uploading piece. err: ecclient: upload failed (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, address:45.91.116.6:4973): protocol: write tcp 188.166.225.170:59438->45.91.116.6:4973: use of closed network connection; write tcp 188.166.225.170:59438->45.91.116.6:4973: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59438->45.91.116.6:4973: use of closed network connection Aug 9 11:37:54.124840752 (18h11m48.555920883s) [noise=true , address=77.90.31.17:4772 , piece_id=MGZPHOODRIOTGABQJHNIP5FGF4XN6A723ZHUBKI6KNAJDKFWWOJA, seg_pos={0 0}, upload=18998]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: write tcp 188.166.225.170:42704->77.90.31.17:4772: use of closed network connection; write tcp 188.166.225.170:42704->77.90.31.17:4772: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42704->77.90.31.17:4772: use of closed network connection Aug 9 11:37:54.124845251 (18h11m48.555925382s) [noise=true , address=45.91.116.8:4047 , piece_id=B27B2QKZXICP4DHTDTCYSZZ7RN2PPCMKSRQBKRK5JJ4BE4RNWQAQ, seg_pos={0 0}, upload=18998]: Done uploading piece. err: ecclient: upload failed (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN, address:45.91.116.8:4047): protocol: write tcp 188.166.225.170:40794->45.91.116.8:4047: use of closed network connection; write tcp 188.166.225.170:40794->45.91.116.8:4047: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40794->45.91.116.8:4047: use of closed network connection Aug 9 11:37:54.124866321 (18h11m48.555946492s) [seg_pos={0 0}, upload=18998]: Done waiting for segment. successful: 6 optimal: 13 errs: <nil> Aug 9 11:37:54.353990811 (18h11m48.785070942s) [upload=18998]: Done uploading segments... Aug 9 11:37:54.354028673 (18h11m48.785108804s) [upload=18998]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 6) Duration: 5.972505297s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 3 out of 60 is failed (threshold=3) [.........................................X............X.X...] duration: 6.019479155s FAIL (2025-08-09 12:03:03 UTC) orbiter warning: operation: ListObjects ======ERROR====== unable to find key in bucket: sj://uptime-checks-placement-20/orbiter_us_0/4M ======LOGS====== Aug 9 12:00:23.108483377 (18h34m17.539563508s) [upload=19402]: Starting upload Aug 9 12:00:23.108636078 (18h34m17.53971621s) [upload=19402]: Done starting upload Aug 9 12:00:23.108746840 (18h34m17.539826981s) [upload=19402]: Uploading segments... Aug 9 12:00:23.108904442 (18h34m17.539984573s) [upload=19402]: Got next segment. Inline: false Aug 9 12:00:23.108970418 (18h34m17.540050599s) [upload=19402]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 9 12:00:23.108977743 (18h34m17.540057904s) [upload=19402]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 9 12:00:23.579075368 (18h34m18.010155499s) [seg_pos={0 0}, upload=19402]: Begin upload segment... Aug 9 12:00:23.579127947 (18h34m18.010208078s) [seg_pos={0 0}, upload=19402]: Done begin upload segment. Aug 9 12:00:23.579157074 (18h34m18.010237205s) [noise=true , address=104.36.82.44:4899 , piece_id=KIQ5AFZ2FIUJ3XIG5ZQBJYITK5CD4ZDW5SYOUPFIYAOZBPENWFFA, seg_pos={0 0}, upload=19402]: Uploading piece... Aug 9 12:00:23.579206939 (18h34m18.01028707s) [noise=true , address=104.36.82.55:4199 , piece_id=6D4WT6KDBSMMCPXONV74YG4IXFEXPXU2FQTCL2W2NPZ2ERVMBUMQ, seg_pos={0 0}, upload=19402]: Uploading piece... Aug 9 12:00:23.579552951 (18h34m18.010633082s) [noise=true , address=64.49.13.139:4469 , piece_id=UTZT7WZ3LQ6LOZ3C6IXV5M7L3APATJOYDRZAMWUEYCZP4I34THCA, seg_pos={0 0}, upload=19402]: Uploading piece... Aug 9 12:00:23.579921666 (18h34m18.011001797s) [noise=true , address=104.36.82.59:4327 , piece_id=WJJFOV44IRLH2E6BFC46ALBJXD54ML4HQOFSUDFNJ3OJUXASSTZQ, seg_pos={0 0}, upload=19402]: Uploading piece... Aug 9 12:00:23.580275103 (18h34m18.011355234s) [noise=true , address=104.36.82.13:4197 , piece_id=NEMRRYQPHXM73GSZVEETU7UNMUDQVNKKSLKQTRSBHBKPQ35KLLPQ, seg_pos={0 0}, upload=19402]: Uploading piece... Aug 9 12:00:23.580699144 (18h34m18.011779275s) [noise=true , address=104.36.82.23:4454 , piece_id=VNKC653W6NNZTTH6CRDK2LSD5XFLLGJGYOOIELABDCKC364DITDA, seg_pos={0 0}, upload=19402]: Uploading piece... Aug 9 12:00:23.580803934 (18h34m18.011884065s) [noise=true , address=104.36.82.61:4396 , piece_id=LJ73GMLI4ZDNLDSGIRT4GRBU56YFUOBTOC7OW3NS4TRWSUXMQCHA, seg_pos={0 0}, upload=19402]: Uploading piece... Aug 9 12:00:23.581071286 (18h34m18.012151417s) [noise=true , address=104.36.82.45:4934 , piece_id=K7XWI37JOPKYII6BFSO6AZT73S3FYOH5JKCJXOM5O5SCWAG62OIQ, seg_pos={0 0}, upload=19402]: Uploading piece... Aug 9 12:00:23.581305595 (18h34m18.012385726s) [noise=true , address=104.36.82.60:4354 , piece_id=HR52C2IWZNCL2XI553EUKTS37R5N4ILVW34GJSPWJV3Z3W7KKGWA, seg_pos={0 0}, upload=19402]: Uploading piece... Aug 9 12:00:23.581541305 (18h34m18.012621436s) [noise=true , address=45.91.116.12:4243 , piece_id=FZW53SVI3PFJ3M6WQE74FPB737JEBGBGA2YB53XCU5DEPEUQHU7Q, seg_pos={0 0}, upload=19402]: Uploading piece... Aug 9 12:00:23.581620086 (18h34m18.012700217s) [noise=true , address=104.36.82.58:4268 , piece_id=QLMKG6PZ4XYJJUTOHFW7EWTFGTST5HYLTHP5MMX6X3AJJJAZSZ3Q, seg_pos={0 0}, upload=19402]: Uploading piece... Aug 9 12:00:23.581922384 (18h34m18.013002515s) [noise=true , address=104.36.82.9:4031 , piece_id=LLSOV26QMVJRDNGWSOKPT4MIGAMTHU6AIADGGJNG6526XAR3SF6Q, seg_pos={0 0}, upload=19402]: Uploading piece... Aug 9 12:00:23.582143177 (18h34m18.013223308s) [noise=true , address=104.36.82.60:4358 , piece_id=WPW7FORNXWY6DOAXVH775D4W3VFOH7PRMQO5UXKJNIFFCRVSE43A, seg_pos={0 0}, upload=19402]: Uploading piece... Aug 9 12:00:23.582347538 (18h34m18.013427669s) [noise=true , address=104.36.82.14:4242 , piece_id=2JREAT5B4ZOSHBJPMPKVQOW2VDMLQIDURO7KTYRPFJAM6U5K57WQ, seg_pos={0 0}, upload=19402]: Uploading piece... Aug 9 12:00:23.582596004 (18h34m18.013676135s) [noise=true , address=64.49.13.170:4726 , piece_id=A2OWN4ZBVO3MZKLRAHERFMYAZDSLB4HMYESALCEVEH66LGUZEQOQ, seg_pos={0 0}, upload=19402]: Uploading piece... Aug 9 12:00:23.582774395 (18h34m18.013854526s) [noise=true , address=104.36.82.46:4484 , piece_id=FQHBXDSLXK2GIOSY4V5QBBPLOH7ZD3KHJVXR5IFYPMICQCWH7UFA, seg_pos={0 0}, upload=19402]: Uploading piece... Aug 9 12:00:23.583063899 (18h34m18.01414403s) [noise=true , address=77.90.31.12:4604 , piece_id=HPMFA765VZOJGNXPWE5SCA4AAK7QB4HFGUSYFQWOFLVZTU6PSUMA, seg_pos={0 0}, upload=19402]: Uploading piece... Aug 9 12:00:23.583138481 (18h34m18.014218612s) [noise=true , address=104.36.82.5:4845 , piece_id=5AJNZJD6MFKE3U6EZ27IEXQEO4KOUGSXA7EBCDSWVTVFYD4AJTHQ, seg_pos={0 0}, upload=19402]: Uploading piece... Aug 9 12:00:24.683477824 (18h34m19.114557955s) [upload=19402]: Next returned nil segment Aug 9 12:00:24.683489506 (18h34m19.114569637s) [upload=19402]: Waiting for error group managing segments... Aug 9 12:00:24.955322603 (18h34m19.386402734s) [noise=true , address=64.49.13.170:4726 , piece_id=A2OWN4ZBVO3MZKLRAHERFMYAZDSLB4HMYESALCEVEH66LGUZEQOQ, seg_pos={0 0}, upload=19402]: Done uploading piece. err: <nil> Aug 9 12:00:24.969832404 (18h34m19.400912535s) [noise=true , address=64.49.13.139:4469 , piece_id=UTZT7WZ3LQ6LOZ3C6IXV5M7L3APATJOYDRZAMWUEYCZP4I34THCA, seg_pos={0 0}, upload=19402]: Done uploading piece. err: <nil> Aug 9 12:00:25.516280300 (18h34m19.947360431s) [noise=true , address=104.36.82.55:4199 , piece_id=6D4WT6KDBSMMCPXONV74YG4IXFEXPXU2FQTCL2W2NPZ2ERVMBUMQ, seg_pos={0 0}, upload=19402]: Done uploading piece. err: <nil> Aug 9 12:00:25.525386162 (18h34m19.956466293s) [noise=true , address=104.36.82.46:4484 , piece_id=FQHBXDSLXK2GIOSY4V5QBBPLOH7ZD3KHJVXR5IFYPMICQCWH7UFA, seg_pos={0 0}, upload=19402]: Done uploading piece. err: <nil> Aug 9 12:00:25.526709204 (18h34m19.957789335s) [noise=true , address=104.36.82.60:4358 , piece_id=WPW7FORNXWY6DOAXVH775D4W3VFOH7PRMQO5UXKJNIFFCRVSE43A, seg_pos={0 0}, upload=19402]: Done uploading piece. err: <nil> Aug 9 12:00:25.530790893 (18h34m19.961871024s) [noise=true , address=104.36.82.44:4899 , piece_id=KIQ5AFZ2FIUJ3XIG5ZQBJYITK5CD4ZDW5SYOUPFIYAOZBPENWFFA, seg_pos={0 0}, upload=19402]: Done uploading piece. err: <nil> Aug 9 12:00:25.531900687 (18h34m19.962980818s) [noise=true , address=104.36.82.58:4268 , piece_id=QLMKG6PZ4XYJJUTOHFW7EWTFGTST5HYLTHP5MMX6X3AJJJAZSZ3Q, seg_pos={0 0}, upload=19402]: Done uploading piece. err: <nil> Aug 9 12:00:25.536797967 (18h34m19.967878098s) [noise=true , address=104.36.82.9:4031 , piece_id=LLSOV26QMVJRDNGWSOKPT4MIGAMTHU6AIADGGJNG6526XAR3SF6Q, seg_pos={0 0}, upload=19402]: Done uploading piece. err: <nil> Aug 9 12:00:25.545685641 (18h34m19.976765772s) [noise=true , address=104.36.82.60:4354 , piece_id=HR52C2IWZNCL2XI553EUKTS37R5N4ILVW34GJSPWJV3Z3W7KKGWA, seg_pos={0 0}, upload=19402]: Done uploading piece. err: <nil> Aug 9 12:00:25.555951362 (18h34m19.987031493s) [noise=true , address=104.36.82.61:4396 , piece_id=LJ73GMLI4ZDNLDSGIRT4GRBU56YFUOBTOC7OW3NS4TRWSUXMQCHA, seg_pos={0 0}, upload=19402]: Done uploading piece. err: <nil> Aug 9 12:00:25.566816712 (18h34m19.997896843s) [noise=true , address=104.36.82.5:4845 , piece_id=5AJNZJD6MFKE3U6EZ27IEXQEO4KOUGSXA7EBCDSWVTVFYD4AJTHQ, seg_pos={0 0}, upload=19402]: Done uploading piece. err: <nil> Aug 9 12:00:25.585861129 (18h34m20.01694126s) [noise=true , address=104.36.82.45:4934 , piece_id=K7XWI37JOPKYII6BFSO6AZT73S3FYOH5JKCJXOM5O5SCWAG62OIQ, seg_pos={0 0}, upload=19402]: Done uploading piece. err: <nil> Aug 9 12:00:25.594196727 (18h34m20.025276858s) [noise=true , address=45.91.116.12:4243 , piece_id=FZW53SVI3PFJ3M6WQE74FPB737JEBGBGA2YB53XCU5DEPEUQHU7Q, seg_pos={0 0}, upload=19402]: Done uploading piece. err: <nil> Aug 9 12:00:25.606123457 (18h34m20.037203588s) [noise=true , address=77.90.31.12:4604 , piece_id=HPMFA765VZOJGNXPWE5SCA4AAK7QB4HFGUSYFQWOFLVZTU6PSUMA, seg_pos={0 0}, upload=19402]: Done uploading piece. err: <nil> Aug 9 12:00:25.606140419 (18h34m20.03722055s) [seg_pos={0 0}, upload=19402]: Segment reached optimal threshold of 14 pieces. Aug 9 12:00:25.606241462 (18h34m20.037321593s) [noise=true , address=104.36.82.13:4197 , piece_id=NEMRRYQPHXM73GSZVEETU7UNMUDQVNKKSLKQTRSBHBKPQ35KLLPQ, seg_pos={0 0}, upload=19402]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1cB89C5xnya9WGD9dGBsSouqkZH3nLqytrg9Qo4UH2kdFonTjM): protocol: expected piece hash; context canceled; EOF Aug 9 12:00:25.606346553 (18h34m20.037426675s) [noise=true , address=104.36.82.14:4242 , piece_id=2JREAT5B4ZOSHBJPMPKVQOW2VDMLQIDURO7KTYRPFJAM6U5K57WQ, seg_pos={0 0}, upload=19402]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1inR8rjUp4fRQWcaD2S7MRxveHyFxKoutWoEiuKQG3C895PaMz): protocol: expected piece hash; context canceled; EOF Aug 9 12:00:25.606405596 (18h34m20.037485727s) [noise=true , address=104.36.82.23:4454 , piece_id=VNKC653W6NNZTTH6CRDK2LSD5XFLLGJGYOOIELABDCKC364DITDA, seg_pos={0 0}, upload=19402]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1UCkuYGdVYLoR8oc1jpCZCN2smNfVqFwC8a7q95fJb8haB3MY2): protocol: expected piece hash; context canceled; EOF Aug 9 12:00:25.606463838 (18h34m20.037543969s) [noise=true , address=104.36.82.59:4327 , piece_id=WJJFOV44IRLH2E6BFC46ALBJXD54ML4HQOFSUDFNJ3OJUXASSTZQ, seg_pos={0 0}, upload=19402]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1V4C6AWGbyYQWfinoSj7iKovHeTJ9hqypyC5U9rZ8pveXBBfCY): protocol: expected piece hash; context canceled; EOF Aug 9 12:00:25.606478186 (18h34m20.037558317s) [seg_pos={0 0}, upload=19402]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 9 12:00:25.606535966 (18h34m20.037616097s) [upload=19402]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 9 12:00:25.606540855 (18h34m20.037620976s) [upload=19402]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 9 12:00:25.913625167 (18h34m20.344705298s) [upload=19402]: Done uploading segments... Aug 9 12:00:25.913644565 (18h34m20.344724696s) [upload=19402]: Upload finished. err: <nil> Duration: 6.019505785s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 3 out of 60 is failed (threshold=3) [..........................X..................X.........X....] duration: 2.074523265s FAIL (2025-08-09 12:02:49 UTC) orbiter warning: operation: ListObjects ======ERROR====== unable to find key in bucket: sj://uptime-checks-placement-20/orbiter_us_0/1K ======LOGS====== Aug 9 11:59:31.819436107 (18h33m26.250516238s) [upload=19388]: Starting upload Aug 9 11:59:31.819492355 (18h33m26.250572516s) [upload=19388]: Done starting upload Aug 9 11:59:31.819563080 (18h33m26.250643211s) [upload=19388]: Uploading segments... Aug 9 11:59:31.819616853 (18h33m26.250696984s) [upload=19388]: Got next segment. Inline: true Aug 9 11:59:31.819621552 (18h33m26.250701683s) [upload=19388]: Waiting for error group managing segments... Aug 9 11:59:31.819660466 (18h33m26.250740597s) [upload=19388]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 9 11:59:31.819665206 (18h33m26.250745327s) [upload=19388]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 9 11:59:31.819667851 (18h33m26.250747972s) [upload=19388]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 9 11:59:32.242597809 (18h33m26.67367794s) [upload=19388]: Done uploading segments... Aug 9 11:59:32.242615142 (18h33m26.673695263s) [upload=19388]: Upload finished. err: <nil> Duration:* 2.074546349s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 3 out of 60 is failed (threshold=3) [..........................................X............X.X..] duration: 5.881496271s FAIL (2025-08-09 12:02:39 UTC) orbiter warning: operation: ListObjects ======ERROR====== unable to find key in bucket: sj://uptime-checks-placement-20/orbiter_us_0/4M ======LOGS====== Aug 9 12:00:23.108483377 (18h34m17.539563508s) [upload=19402]: Starting upload Aug 9 12:00:23.108636078 (18h34m17.53971621s) [upload=19402]: Done starting upload Aug 9 12:00:23.108746840 (18h34m17.539826981s) [upload=19402]: Uploading segments... Aug 9 12:00:23.108904442 (18h34m17.539984573s) [upload=19402]: Got next segment. Inline: false Aug 9 12:00:23.108970418 (18h34m17.540050599s) [upload=19402]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 9 12:00:23.108977743 (18h34m17.540057904s) [upload=19402]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 9 12:00:23.579075368 (18h34m18.010155499s) [seg_pos={0 0}, upload=19402]: Begin upload segment... Aug 9 12:00:23.579127947 (18h34m18.010208078s) [seg_pos={0 0}, upload=19402]: Done begin upload segment. Aug 9 12:00:23.579157074 (18h34m18.010237205s) [noise=true , address=104.36.82.44:4899 , piece_id=KIQ5AFZ2FIUJ3XIG5ZQBJYITK5CD4ZDW5SYOUPFIYAOZBPENWFFA, seg_pos={0 0}, upload=19402]: Uploading piece... Aug 9 12:00:23.579206939 (18h34m18.01028707s) [noise=true , address=104.36.82.55:4199 , piece_id=6D4WT6KDBSMMCPXONV74YG4IXFEXPXU2FQTCL2W2NPZ2ERVMBUMQ, seg_pos={0 0}, upload=19402]: Uploading piece... Aug 9 12:00:23.579552951 (18h34m18.010633082s) [noise=true , address=64.49.13.139:4469 , piece_id=UTZT7WZ3LQ6LOZ3C6IXV5M7L3APATJOYDRZAMWUEYCZP4I34THCA, seg_pos={0 0}, upload=19402]: Uploading piece... Aug 9 12:00:23.579921666 (18h34m18.011001797s) [noise=true , address=104.36.82.59:4327 , piece_id=WJJFOV44IRLH2E6BFC46ALBJXD54ML4HQOFSUDFNJ3OJUXASSTZQ, seg_pos={0 0}, upload=19402]: Uploading piece... Aug 9 12:00:23.580275103 (18h34m18.011355234s) [noise=true , address=104.36.82.13:4197 , piece_id=NEMRRYQPHXM73GSZVEETU7UNMUDQVNKKSLKQTRSBHBKPQ35KLLPQ, seg_pos={0 0}, upload=19402]: Uploading piece... Aug 9 12:00:23.580699144 (18h34m18.011779275s) [noise=true , address=104.36.82.23:4454 , piece_id=VNKC653W6NNZTTH6CRDK2LSD5XFLLGJGYOOIELABDCKC364DITDA, seg_pos={0 0}, upload=19402]: Uploading piece... Aug 9 12:00:23.580803934 (18h34m18.011884065s) [noise=true , address=104.36.82.61:4396 , piece_id=LJ73GMLI4ZDNLDSGIRT4GRBU56YFUOBTOC7OW3NS4TRWSUXMQCHA, seg_pos={0 0}, upload=19402]: Uploading piece... Aug 9 12:00:23.581071286 (18h34m18.012151417s) [noise=true , address=104.36.82.45:4934 , piece_id=K7XWI37JOPKYII6BFSO6AZT73S3FYOH5JKCJXOM5O5SCWAG62OIQ, seg_pos={0 0}, upload=19402]: Uploading piece... Aug 9 12:00:23.581305595 (18h34m18.012385726s) [noise=true , address=104.36.82.60:4354 , piece_id=HR52C2IWZNCL2XI553EUKTS37R5N4ILVW34GJSPWJV3Z3W7KKGWA, seg_pos={0 0}, upload=19402]: Uploading piece... Aug 9 12:00:23.581541305 (18h34m18.012621436s) [noise=true , address=45.91.116.12:4243 , piece_id=FZW53SVI3PFJ3M6WQE74FPB737JEBGBGA2YB53XCU5DEPEUQHU7Q, seg_pos={0 0}, upload=19402]: Uploading piece... Aug 9 12:00:23.581620086 (18h34m18.012700217s) [noise=true , address=104.36.82.58:4268 , piece_id=QLMKG6PZ4XYJJUTOHFW7EWTFGTST5HYLTHP5MMX6X3AJJJAZSZ3Q, seg_pos={0 0}, upload=19402]: Uploading piece... Aug 9 12:00:23.581922384 (18h34m18.013002515s) [noise=true , address=104.36.82.9:4031 , piece_id=LLSOV26QMVJRDNGWSOKPT4MIGAMTHU6AIADGGJNG6526XAR3SF6Q, seg_pos={0 0}, upload=19402]: Uploading piece... Aug 9 12:00:23.582143177 (18h34m18.013223308s) [noise=true , address=104.36.82.60:4358 , piece_id=WPW7FORNXWY6DOAXVH775D4W3VFOH7PRMQO5UXKJNIFFCRVSE43A, seg_pos={0 0}, upload=19402]: Uploading piece... Aug 9 12:00:23.582347538 (18h34m18.013427669s) [noise=true , address=104.36.82.14:4242 , piece_id=2JREAT5B4ZOSHBJPMPKVQOW2VDMLQIDURO7KTYRPFJAM6U5K57WQ, seg_pos={0 0}, upload=19402]: Uploading piece... Aug 9 12:00:23.582596004 (18h34m18.013676135s) [noise=true , address=64.49.13.170:4726 , piece_id=A2OWN4ZBVO3MZKLRAHERFMYAZDSLB4HMYESALCEVEH66LGUZEQOQ, seg_pos={0 0}, upload=19402]: Uploading piece... Aug 9 12:00:23.582774395 (18h34m18.013854526s) [noise=true , address=104.36.82.46:4484 , piece_id=FQHBXDSLXK2GIOSY4V5QBBPLOH7ZD3KHJVXR5IFYPMICQCWH7UFA, seg_pos={0 0}, upload=19402]: Uploading piece... Aug 9 12:00:23.583063899 (18h34m18.01414403s) [noise=true , address=77.90.31.12:4604 , piece_id=HPMFA765VZOJGNXPWE5SCA4AAK7QB4HFGUSYFQWOFLVZTU6PSUMA, seg_pos={0 0}, upload=19402]: Uploading piece... Aug 9 12:00:23.583138481 (18h34m18.014218612s) [noise=true , address=104.36.82.5:4845 , piece_id=5AJNZJD6MFKE3U6EZ27IEXQEO4KOUGSXA7EBCDSWVTVFYD4AJTHQ, seg_pos={0 0}, upload=19402]: Uploading piece... Aug 9 12:00:24.683477824 (18h34m19.114557955s) [upload=19402]: Next returned nil segment Aug 9 12:00:24.683489506 (18h34m19.114569637s) [upload=19402]: Waiting for error group managing segments... Aug 9 12:00:24.955322603 (18h34m19.386402734s) [noise=true , address=64.49.13.170:4726 , piece_id=A2OWN4ZBVO3MZKLRAHERFMYAZDSLB4HMYESALCEVEH66LGUZEQOQ, seg_pos={0 0}, upload=19402]: Done uploading piece. err: <nil> Aug 9 12:00:24.969832404 (18h34m19.400912535s) [noise=true , address=64.49.13.139:4469 , piece_id=UTZT7WZ3LQ6LOZ3C6IXV5M7L3APATJOYDRZAMWUEYCZP4I34THCA, seg_pos={0 0}, upload=19402]: Done uploading piece. err: <nil> Aug 9 12:00:25.516280300 (18h34m19.947360431s) [noise=true , address=104.36.82.55:4199 , piece_id=6D4WT6KDBSMMCPXONV74YG4IXFEXPXU2FQTCL2W2NPZ2ERVMBUMQ, seg_pos={0 0}, upload=19402]: Done uploading piece. err: <nil> Aug 9 12:00:25.525386162 (18h34m19.956466293s) [noise=true , address=104.36.82.46:4484 , piece_id=FQHBXDSLXK2GIOSY4V5QBBPLOH7ZD3KHJVXR5IFYPMICQCWH7UFA, seg_pos={0 0}, upload=19402]: Done uploading piece. err: <nil> Aug 9 12:00:25.526709204 (18h34m19.957789335s) [noise=true , address=104.36.82.60:4358 , piece_id=WPW7FORNXWY6DOAXVH775D4W3VFOH7PRMQO5UXKJNIFFCRVSE43A, seg_pos={0 0}, upload=19402]: Done uploading piece. err: <nil> Aug 9 12:00:25.530790893 (18h34m19.961871024s) [noise=true , address=104.36.82.44:4899 , piece_id=KIQ5AFZ2FIUJ3XIG5ZQBJYITK5CD4ZDW5SYOUPFIYAOZBPENWFFA, seg_pos={0 0}, upload=19402]: Done uploading piece. err: <nil> Aug 9 12:00:25.531900687 (18h34m19.962980818s) [noise=true , address=104.36.82.58:4268 , piece_id=QLMKG6PZ4XYJJUTOHFW7EWTFGTST5HYLTHP5MMX6X3AJJJAZSZ3Q, seg_pos={0 0}, upload=19402]: Done uploading piece. err: <nil> Aug 9 12:00:25.536797967 (18h34m19.967878098s) [noise=true , address=104.36.82.9:4031 , piece_id=LLSOV26QMVJRDNGWSOKPT4MIGAMTHU6AIADGGJNG6526XAR3SF6Q, seg_pos={0 0}, upload=19402]: Done uploading piece. err: <nil> Aug 9 12:00:25.545685641 (18h34m19.976765772s) [noise=true , address=104.36.82.60:4354 , piece_id=HR52C2IWZNCL2XI553EUKTS37R5N4ILVW34GJSPWJV3Z3W7KKGWA, seg_pos={0 0}, upload=19402]: Done uploading piece. err: <nil> Aug 9 12:00:25.555951362 (18h34m19.987031493s) [noise=true , address=104.36.82.61:4396 , piece_id=LJ73GMLI4ZDNLDSGIRT4GRBU56YFUOBTOC7OW3NS4TRWSUXMQCHA, seg_pos={0 0}, upload=19402]: Done uploading piece. err: <nil> Aug 9 12:00:25.566816712 (18h34m19.997896843s) [noise=true , address=104.36.82.5:4845 , piece_id=5AJNZJD6MFKE3U6EZ27IEXQEO4KOUGSXA7EBCDSWVTVFYD4AJTHQ, seg_pos={0 0}, upload=19402]: Done uploading piece. err: <nil> Aug 9 12:00:25.585861129 (18h34m20.01694126s) [noise=true , address=104.36.82.45:4934 , piece_id=K7XWI37JOPKYII6BFSO6AZT73S3FYOH5JKCJXOM5O5SCWAG62OIQ, seg_pos={0 0}, upload=19402]: Done uploading piece. err: <nil> Aug 9 12:00:25.594196727 (18h34m20.025276858s) [noise=true , address=45.91.116.12:4243 , piece_id=FZW53SVI3PFJ3M6WQE74FPB737JEBGBGA2YB53XCU5DEPEUQHU7Q, seg_pos={0 0}, upload=19402]: Done uploading piece. err: <nil> Aug 9 12:00:25.606123457 (18h34m20.037203588s) [noise=true , address=77.90.31.12:4604 , piece_id=HPMFA765VZOJGNXPWE5SCA4AAK7QB4HFGUSYFQWOFLVZTU6PSUMA, seg_pos={0 0}, upload=19402]: Done uploading piece. err: <nil> Aug 9 12:00:25.606140419 (18h34m20.03722055s) [seg_pos={0 0}, upload=19402]: Segment reached optimal threshold of 14 pieces. Aug 9 12:00:25.606241462 (18h34m20.037321593s) [noise=true , address=104.36.82.13:4197 , piece_id=NEMRRYQPHXM73GSZVEETU7UNMUDQVNKKSLKQTRSBHBKPQ35KLLPQ, seg_pos={0 0}, upload=19402]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1cB89C5xnya9WGD9dGBsSouqkZH3nLqytrg9Qo4UH2kdFonTjM): protocol: expected piece hash; context canceled; EOF Aug 9 12:00:25.606346553 (18h34m20.037426675s) [noise=true , address=104.36.82.14:4242 , piece_id=2JREAT5B4ZOSHBJPMPKVQOW2VDMLQIDURO7KTYRPFJAM6U5K57WQ, seg_pos={0 0}, upload=19402]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1inR8rjUp4fRQWcaD2S7MRxveHyFxKoutWoEiuKQG3C895PaMz): protocol: expected piece hash; context canceled; EOF Aug 9 12:00:25.606405596 (18h34m20.037485727s) [noise=true , address=104.36.82.23:4454 , piece_id=VNKC653W6NNZTTH6CRDK2LSD5XFLLGJGYOOIELABDCKC364DITDA, seg_pos={0 0}, upload=19402]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1UCkuYGdVYLoR8oc1jpCZCN2smNfVqFwC8a7q95fJb8haB3MY2): protocol: expected piece hash; context canceled; EOF Aug 9 12:00:25.606463838 (18h34m20.037543969s) [noise=true , address=104.36.82.59:4327 , piece_id=WJJFOV44IRLH2E6BFC46ALBJXD54ML4HQOFSUDFNJ3OJUXASSTZQ, seg_pos={0 0}, upload=19402]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1V4C6AWGbyYQWfinoSj7iKovHeTJ9hqypyC5U9rZ8pveXBBfCY): protocol: expected piece hash; context canceled; EOF Aug 9 12:00:25.606478186 (18h34m20.037558317s) [seg_pos={0 0}, upload=19402]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 9 12:00:25.606535966 (18h34m20.037616097s) [upload=19402]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 9 12:00:25.606540855 (18h34m20.037620976s) [upload=19402]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 9 12:00:25.913625167 (18h34m20.344705298s) [upload=19402]: Done uploading segments... Aug 9 12:00:25.913644565 (18h34m20.344724696s) [upload=19402]: Upload finished. err: <nil> Duration: 5.881528121s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................X................................] duration: 5.740154664s FAIL (2025-08-09 12:02:16 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 6) ======LOGS====== ADHZNIB6LBP5VQHFEGSYM3YNNZWNM2N4UU7JTCQ, seg_pos={0 0}, upload=18998]: Uploading piece... Aug 9 11:37:25.005421531 (18h11m19.436501662s) [noise=true , address=45.91.116.4:4940 , piece_id=M33ZGUCEF23HDGQZE5F5KLWQ43NHC7E7LKOVVGQ6P6TM2HFGB4WQ, seg_pos={0 0}, upload=18998]: Uploading piece... Aug 9 11:37:25.005614329 (18h11m19.43669446s) [noise=true , address=45.91.116.8:4047 , piece_id=B27B2QKZXICP4DHTDTCYSZZ7RN2PPCMKSRQBKRK5JJ4BE4RNWQAQ, seg_pos={0 0}, upload=18998]: Uploading piece... Aug 9 11:37:25.005727536 (18h11m19.436807667s) [noise=true , address=45.91.116.12:4225 , piece_id=VFK2SMVTBT76D7YLQWGRA6TF47JD3B7J7F5NFR4E2MQVNGNHYTAQ, seg_pos={0 0}, upload=18998]: Uploading piece... Aug 9 11:37:25.006281517 (18h11m19.437361648s) [noise=true , address=104.36.82.7:4909 , piece_id=OGG5ZCXSLIM5YKMM7K6KLKLWGOTX376DUTDT3567OB3AJIBPBKDQ, seg_pos={0 0}, upload=18998]: Uploading piece... Aug 9 11:37:25.006632558 (18h11m19.437712679s) [noise=true , address=45.91.116.13:4155 , piece_id=K4CGD4OCR2B6ZWAZHB5MDBECVJB3AC2HDXS2TUVAVGIWQYJEARLA, seg_pos={0 0}, upload=18998]: Uploading piece... Aug 9 11:37:25.006996264 (18h11m19.438076395s) [noise=true , address=77.90.31.18:4806 , piece_id=QCURQYEJL22LBT2YFBR62RIHH7VNSGL7GERP66GKHXK3JP5WLVZQ, seg_pos={0 0}, upload=18998]: Uploading piece... Aug 9 11:37:25.007245231 (18h11m19.438325362s) [noise=true , address=104.36.82.3:4765 , piece_id=HBLTTPZYK5ZXNG2XV53QVJA3NPX3YWKEAAAFVMDXUO6BLR3QKCWQ, seg_pos={0 0}, upload=18998]: Uploading piece... Aug 9 11:37:25.007475481 (18h11m19.438555612s) [noise=true , address=45.91.116.6:4973 , piece_id=MHDNXTVKFMLRKGS7JIPL4U5VQMBXGP4EYM7JEGPSK4U3GIJ5JEHQ, seg_pos={0 0}, upload=18998]: Uploading piece... Aug 9 11:37:25.007873984 (18h11m19.438954115s) [noise=true , address=45.91.116.7:4010 , piece_id=5G3O552GD5XO7SNIHST623QJBWJF2C7B6EN7ISGHHY3FYV6QL2DQ, seg_pos={0 0}, upload=18998]: Uploading piece... Aug 9 11:37:26.733124951 (18h11m21.164205082s) [upload=18998]: Next returned nil segment Aug 9 11:37:26.733142194 (18h11m21.164222325s) [upload=18998]: Waiting for error group managing segments... Aug 9 11:37:27.100757697 (18h11m21.531837848s) [noise=true , address=104.36.82.14:4222 , piece_id=Z4X6RJ5HBAYGUNKT6DXWSS2TEG2OUFPMH3NWYGA7D26XIFH46UAA, seg_pos={0 0}, upload=18998]: Done uploading piece. err: <nil> Aug 9 11:37:27.106781913 (18h11m21.537862064s) [noise=true , address=104.36.82.7:4912 , piece_id=MRA5X6N6MDQ2TADHZNIB6LBP5VQHFEGSYM3YNNZWNM2N4UU7JTCQ, seg_pos={0 0}, upload=18998]: Done uploading piece. err: <nil> Aug 9 11:37:27.136679105 (18h11m21.567759236s) [noise=true , address=104.36.82.5:4838 , piece_id=T3WCWCLB4WAXQY3S6VCFHLKYK6IY2Q6P2776CLGYS4QOFTXMYVEQ, seg_pos={0 0}, upload=18998]: Done uploading piece. err: <nil> Aug 9 11:37:27.139126447 (18h11m21.570206578s) [noise=true , address=104.36.82.3:4765 , piece_id=HBLTTPZYK5ZXNG2XV53QVJA3NPX3YWKEAAAFVMDXUO6BLR3QKCWQ, seg_pos={0 0}, upload=18998]: Done uploading piece. err: <nil> Aug 9 11:37:27.183514474 (18h11m21.614594606s) [noise=true , address=104.36.82.7:4909 , piece_id=OGG5ZCXSLIM5YKMM7K6KLKLWGOTX376DUTDT3567OB3AJIBPBKDQ, seg_pos={0 0}, upload=18998]: Done uploading piece. err: <nil> Aug 9 11:37:53.489259032 (18h11m47.920339163s) [noise=true , address=45.91.116.13:4155 , piece_id=K4CGD4OCR2B6ZWAZHB5MDBECVJB3AC2HDXS2TUVAVGIWQYJEARLA, seg_pos={0 0}, upload=18998]: Done uploading piece. err: <nil> Aug 9 11:37:54.124525138 (18h11m48.555605259s) [noise=true , address=45.91.116.7:4010 , piece_id=5G3O552GD5XO7SNIHST623QJBWJF2C7B6EN7ISGHHY3FYV6QL2DQ, seg_pos={0 0}, upload=18998]: Done uploading piece. err: ecclient: upload failed (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w, address:45.91.116.7:4010): protocol: write tcp 188.166.225.170:54960->45.91.116.7:4010: use of closed network connection; write tcp 188.166.225.170:54960->45.91.116.7:4010: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54960->45.91.116.7:4010: use of closed network connection Aug 9 11:37:54.124581035 (18h11m48.555661166s) [noise=true , address=77.90.31.10:4520 , piece_id=YRBBWM54FGU3B74JQ7EOAO7VSGVFXAVT4SNZ6KN3IQFIHIEVV5HA, seg_pos={0 0}, upload=18998]: Done uploading piece. err: ecclient: upload failed (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp, address:77.90.31.10:4520): protocol: write tcp 188.166.225.170:39560->77.90.31.10:4520: use of closed network connection; write tcp 188.166.225.170:39560->77.90.31.10:4520: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39560->77.90.31.10:4520: use of closed network connection Aug 9 11:37:54.124615480 (18h11m48.555695652s) [noise=true , address=45.91.116.12:4225 , piece_id=VFK2SMVTBT76D7YLQWGRA6TF47JD3B7J7F5NFR4E2MQVNGNHYTAQ, seg_pos={0 0}, upload=18998]: Done uploading piece. err: ecclient: upload failed (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy, address:45.91.116.12:4225): protocol: write tcp 188.166.225.170:38054->45.91.116.12:4225: use of closed network connection; write tcp 188.166.225.170:38054->45.91.116.12:4225: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38054->45.91.116.12:4225: use of closed network connection Aug 9 11:37:54.124643875 (18h11m48.555724016s) [noise=true , address=45.91.116.4:4940 , piece_id=M33ZGUCEF23HDGQZE5F5KLWQ43NHC7E7LKOVVGQ6P6TM2HFGB4WQ, seg_pos={0 0}, upload=18998]: Done uploading piece. err: ecclient: upload failed (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe, address:45.91.116.4:4940): protocol: write tcp 188.166.225.170:40630->45.91.116.4:4940: use of closed network connection; write tcp 188.166.225.170:40630->45.91.116.4:4940: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40630->45.91.116.4:4940: use of closed network connection Aug 9 11:37:54.124693159 (18h11m48.55577335s) [noise=true , address=45.91.116.14:4189 , piece_id=XVQMTYHB6H3IKSERYT2BEB4NP42KV6FRE6J2MKKU5WEOCVBHYIWA, seg_pos={0 0}, upload=18998]: Done uploading piece. err: ecclient: upload failed (node:1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N, address:45.91.116.14:4189): protocol: write tcp 188.166.225.170:37650->45.91.116.14:4189: use of closed network connection; write tcp 188.166.225.170:37650->45.91.116.14:4189: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37650->45.91.116.14:4189: use of closed network connection Aug 9 11:37:54.124771740 (18h11m48.555851871s) [noise=true , address=77.90.31.18:4806 , piece_id=QCURQYEJL22LBT2YFBR62RIHH7VNSGL7GERP66GKHXK3JP5WLVZQ, seg_pos={0 0}, upload=18998]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: write tcp 188.166.225.170:55536->77.90.31.18:4806: use of closed network connection; write tcp 188.166.225.170:55536->77.90.31.18:4806: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55536->77.90.31.18:4806: use of closed network connection Aug 9 11:37:54.124817217 (18h11m48.555897348s) [noise=true , address=77.90.31.15:4697 , piece_id=Q26SRGZO2YBJWPDIG4U422EICE6XEXPLHFHXLKFUBKMMJAZ4W7UQ, seg_pos={0 0}, upload=18998]: Done uploading piece. err: ecclient: upload failed (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, address:77.90.31.15:4697): protocol: write tcp 188.166.225.170:49618->77.90.31.15:4697: use of closed network connection; write tcp 188.166.225.170:49618->77.90.31.15:4697: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49618->77.90.31.15:4697: use of closed network connection Aug 9 11:37:54.124833387 (18h11m48.555913559s) [noise=true , address=45.91.116.6:4973 , piece_id=MHDNXTVKFMLRKGS7JIPL4U5VQMBXGP4EYM7JEGPSK4U3GIJ5JEHQ, seg_pos={0 0}, upload=18998]: Done uploading piece. err: ecclient: upload failed (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, address:45.91.116.6:4973): protocol: write tcp 188.166.225.170:59438->45.91.116.6:4973: use of closed network connection; write tcp 188.166.225.170:59438->45.91.116.6:4973: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59438->45.91.116.6:4973: use of closed network connection Aug 9 11:37:54.124840752 (18h11m48.555920883s) [noise=true , address=77.90.31.17:4772 , piece_id=MGZPHOODRIOTGABQJHNIP5FGF4XN6A723ZHUBKI6KNAJDKFWWOJA, seg_pos={0 0}, upload=18998]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: write tcp 188.166.225.170:42704->77.90.31.17:4772: use of closed network connection; write tcp 188.166.225.170:42704->77.90.31.17:4772: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42704->77.90.31.17:4772: use of closed network connection Aug 9 11:37:54.124845251 (18h11m48.555925382s) [noise=true , address=45.91.116.8:4047 , piece_id=B27B2QKZXICP4DHTDTCYSZZ7RN2PPCMKSRQBKRK5JJ4BE4RNWQAQ, seg_pos={0 0}, upload=18998]: Done uploading piece. err: ecclient: upload failed (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN, address:45.91.116.8:4047): protocol: write tcp 188.166.225.170:40794->45.91.116.8:4047: use of closed network connection; write tcp 188.166.225.170:40794->45.91.116.8:4047: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40794->45.91.116.8:4047: use of closed network connection Aug 9 11:37:54.124866321 (18h11m48.555946492s) [seg_pos={0 0}, upload=18998]: Done waiting for segment. successful: 6 optimal: 13 errs: <nil> Aug 9 11:37:54.353990811 (18h11m48.785070942s) [upload=18998]: Done uploading segments... Aug 9 11:37:54.354028673 (18h11m48.785108804s) [upload=18998]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 6) Duration: 5.740185142s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 3 out of 60 is failed (threshold=3) [...........................X..................X.........X...] duration: 2.006762276s FAIL (2025-08-09 12:02:13 UTC) orbiter warning: operation: ListObjects ======ERROR====== unable to find key in bucket: sj://uptime-checks-placement-20/orbiter_us_0/1K ======LOGS====== Aug 9 11:59:31.819436107 (18h33m26.250516238s) [upload=19388]: Starting upload Aug 9 11:59:31.819492355 (18h33m26.250572516s) [upload=19388]: Done starting upload Aug 9 11:59:31.819563080 (18h33m26.250643211s) [upload=19388]: Uploading segments... Aug 9 11:59:31.819616853 (18h33m26.250696984s) [upload=19388]: Got next segment. Inline: true Aug 9 11:59:31.819621552 (18h33m26.250701683s) [upload=19388]: Waiting for error group managing segments... Aug 9 11:59:31.819660466 (18h33m26.250740597s) [upload=19388]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 9 11:59:31.819665206 (18h33m26.250745327s) [upload=19388]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 9 11:59:31.819667851 (18h33m26.250747972s) [upload=19388]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 9 11:59:32.242597809 (18h33m26.67367794s) [upload=19388]: Done uploading segments... Aug 9 11:59:32.242615142 (18h33m26.673695263s) [upload=19388]: Upload finished. err: <nil> Duration:* 2.006787935s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................X...............................] duration: 5.737527317s FAIL (2025-08-09 12:01:36 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 6) ======LOGS====== ADHZNIB6LBP5VQHFEGSYM3YNNZWNM2N4UU7JTCQ, seg_pos={0 0}, upload=18998]: Uploading piece... Aug 9 11:37:25.005421531 (18h11m19.436501662s) [noise=true , address=45.91.116.4:4940 , piece_id=M33ZGUCEF23HDGQZE5F5KLWQ43NHC7E7LKOVVGQ6P6TM2HFGB4WQ, seg_pos={0 0}, upload=18998]: Uploading piece... Aug 9 11:37:25.005614329 (18h11m19.43669446s) [noise=true , address=45.91.116.8:4047 , piece_id=B27B2QKZXICP4DHTDTCYSZZ7RN2PPCMKSRQBKRK5JJ4BE4RNWQAQ, seg_pos={0 0}, upload=18998]: Uploading piece... Aug 9 11:37:25.005727536 (18h11m19.436807667s) [noise=true , address=45.91.116.12:4225 , piece_id=VFK2SMVTBT76D7YLQWGRA6TF47JD3B7J7F5NFR4E2MQVNGNHYTAQ, seg_pos={0 0}, upload=18998]: Uploading piece... Aug 9 11:37:25.006281517 (18h11m19.437361648s) [noise=true , address=104.36.82.7:4909 , piece_id=OGG5ZCXSLIM5YKMM7K6KLKLWGOTX376DUTDT3567OB3AJIBPBKDQ, seg_pos={0 0}, upload=18998]: Uploading piece... Aug 9 11:37:25.006632558 (18h11m19.437712679s) [noise=true , address=45.91.116.13:4155 , piece_id=K4CGD4OCR2B6ZWAZHB5MDBECVJB3AC2HDXS2TUVAVGIWQYJEARLA, seg_pos={0 0}, upload=18998]: Uploading piece... Aug 9 11:37:25.006996264 (18h11m19.438076395s) [noise=true , address=77.90.31.18:4806 , piece_id=QCURQYEJL22LBT2YFBR62RIHH7VNSGL7GERP66GKHXK3JP5WLVZQ, seg_pos={0 0}, upload=18998]: Uploading piece... Aug 9 11:37:25.007245231 (18h11m19.438325362s) [noise=true , address=104.36.82.3:4765 , piece_id=HBLTTPZYK5ZXNG2XV53QVJA3NPX3YWKEAAAFVMDXUO6BLR3QKCWQ, seg_pos={0 0}, upload=18998]: Uploading piece... Aug 9 11:37:25.007475481 (18h11m19.438555612s) [noise=true , address=45.91.116.6:4973 , piece_id=MHDNXTVKFMLRKGS7JIPL4U5VQMBXGP4EYM7JEGPSK4U3GIJ5JEHQ, seg_pos={0 0}, upload=18998]: Uploading piece... Aug 9 11:37:25.007873984 (18h11m19.438954115s) [noise=true , address=45.91.116.7:4010 , piece_id=5G3O552GD5XO7SNIHST623QJBWJF2C7B6EN7ISGHHY3FYV6QL2DQ, seg_pos={0 0}, upload=18998]: Uploading piece... Aug 9 11:37:26.733124951 (18h11m21.164205082s) [upload=18998]: Next returned nil segment Aug 9 11:37:26.733142194 (18h11m21.164222325s) [upload=18998]: Waiting for error group managing segments... Aug 9 11:37:27.100757697 (18h11m21.531837848s) [noise=true , address=104.36.82.14:4222 , piece_id=Z4X6RJ5HBAYGUNKT6DXWSS2TEG2OUFPMH3NWYGA7D26XIFH46UAA, seg_pos={0 0}, upload=18998]: Done uploading piece. err: <nil> Aug 9 11:37:27.106781913 (18h11m21.537862064s) [noise=true , address=104.36.82.7:4912 , piece_id=MRA5X6N6MDQ2TADHZNIB6LBP5VQHFEGSYM3YNNZWNM2N4UU7JTCQ, seg_pos={0 0}, upload=18998]: Done uploading piece. err: <nil> Aug 9 11:37:27.136679105 (18h11m21.567759236s) [noise=true , address=104.36.82.5:4838 , piece_id=T3WCWCLB4WAXQY3S6VCFHLKYK6IY2Q6P2776CLGYS4QOFTXMYVEQ, seg_pos={0 0}, upload=18998]: Done uploading piece. err: <nil> Aug 9 11:37:27.139126447 (18h11m21.570206578s) [noise=true , address=104.36.82.3:4765 , piece_id=HBLTTPZYK5ZXNG2XV53QVJA3NPX3YWKEAAAFVMDXUO6BLR3QKCWQ, seg_pos={0 0}, upload=18998]: Done uploading piece. err: <nil> Aug 9 11:37:27.183514474 (18h11m21.614594606s) [noise=true , address=104.36.82.7:4909 , piece_id=OGG5ZCXSLIM5YKMM7K6KLKLWGOTX376DUTDT3567OB3AJIBPBKDQ, seg_pos={0 0}, upload=18998]: Done uploading piece. err: <nil> Aug 9 11:37:53.489259032 (18h11m47.920339163s) [noise=true , address=45.91.116.13:4155 , piece_id=K4CGD4OCR2B6ZWAZHB5MDBECVJB3AC2HDXS2TUVAVGIWQYJEARLA, seg_pos={0 0}, upload=18998]: Done uploading piece. err: <nil> Aug 9 11:37:54.124525138 (18h11m48.555605259s) [noise=true , address=45.91.116.7:4010 , piece_id=5G3O552GD5XO7SNIHST623QJBWJF2C7B6EN7ISGHHY3FYV6QL2DQ, seg_pos={0 0}, upload=18998]: Done uploading piece. err: ecclient: upload failed (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w, address:45.91.116.7:4010): protocol: write tcp 188.166.225.170:54960->45.91.116.7:4010: use of closed network connection; write tcp 188.166.225.170:54960->45.91.116.7:4010: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54960->45.91.116.7:4010: use of closed network connection Aug 9 11:37:54.124581035 (18h11m48.555661166s) [noise=true , address=77.90.31.10:4520 , piece_id=YRBBWM54FGU3B74JQ7EOAO7VSGVFXAVT4SNZ6KN3IQFIHIEVV5HA, seg_pos={0 0}, upload=18998]: Done uploading piece. err: ecclient: upload failed (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp, address:77.90.31.10:4520): protocol: write tcp 188.166.225.170:39560->77.90.31.10:4520: use of closed network connection; write tcp 188.166.225.170:39560->77.90.31.10:4520: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39560->77.90.31.10:4520: use of closed network connection Aug 9 11:37:54.124615480 (18h11m48.555695652s) [noise=true , address=45.91.116.12:4225 , piece_id=VFK2SMVTBT76D7YLQWGRA6TF47JD3B7J7F5NFR4E2MQVNGNHYTAQ, seg_pos={0 0}, upload=18998]: Done uploading piece. err: ecclient: upload failed (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy, address:45.91.116.12:4225): protocol: write tcp 188.166.225.170:38054->45.91.116.12:4225: use of closed network connection; write tcp 188.166.225.170:38054->45.91.116.12:4225: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38054->45.91.116.12:4225: use of closed network connection Aug 9 11:37:54.124643875 (18h11m48.555724016s) [noise=true , address=45.91.116.4:4940 , piece_id=M33ZGUCEF23HDGQZE5F5KLWQ43NHC7E7LKOVVGQ6P6TM2HFGB4WQ, seg_pos={0 0}, upload=18998]: Done uploading piece. err: ecclient: upload failed (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe, address:45.91.116.4:4940): protocol: write tcp 188.166.225.170:40630->45.91.116.4:4940: use of closed network connection; write tcp 188.166.225.170:40630->45.91.116.4:4940: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40630->45.91.116.4:4940: use of closed network connection Aug 9 11:37:54.124693159 (18h11m48.55577335s) [noise=true , address=45.91.116.14:4189 , piece_id=XVQMTYHB6H3IKSERYT2BEB4NP42KV6FRE6J2MKKU5WEOCVBHYIWA, seg_pos={0 0}, upload=18998]: Done uploading piece. err: ecclient: upload failed (node:1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N, address:45.91.116.14:4189): protocol: write tcp 188.166.225.170:37650->45.91.116.14:4189: use of closed network connection; write tcp 188.166.225.170:37650->45.91.116.14:4189: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37650->45.91.116.14:4189: use of closed network connection Aug 9 11:37:54.124771740 (18h11m48.555851871s) [noise=true , address=77.90.31.18:4806 , piece_id=QCURQYEJL22LBT2YFBR62RIHH7VNSGL7GERP66GKHXK3JP5WLVZQ, seg_pos={0 0}, upload=18998]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: write tcp 188.166.225.170:55536->77.90.31.18:4806: use of closed network connection; write tcp 188.166.225.170:55536->77.90.31.18:4806: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55536->77.90.31.18:4806: use of closed network connection Aug 9 11:37:54.124817217 (18h11m48.555897348s) [noise=true , address=77.90.31.15:4697 , piece_id=Q26SRGZO2YBJWPDIG4U422EICE6XEXPLHFHXLKFUBKMMJAZ4W7UQ, seg_pos={0 0}, upload=18998]: Done uploading piece. err: ecclient: upload failed (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, address:77.90.31.15:4697): protocol: write tcp 188.166.225.170:49618->77.90.31.15:4697: use of closed network connection; write tcp 188.166.225.170:49618->77.90.31.15:4697: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49618->77.90.31.15:4697: use of closed network connection Aug 9 11:37:54.124833387 (18h11m48.555913559s) [noise=true , address=45.91.116.6:4973 , piece_id=MHDNXTVKFMLRKGS7JIPL4U5VQMBXGP4EYM7JEGPSK4U3GIJ5JEHQ, seg_pos={0 0}, upload=18998]: Done uploading piece. err: ecclient: upload failed (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, address:45.91.116.6:4973): protocol: write tcp 188.166.225.170:59438->45.91.116.6:4973: use of closed network connection; write tcp 188.166.225.170:59438->45.91.116.6:4973: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59438->45.91.116.6:4973: use of closed network connection Aug 9 11:37:54.124840752 (18h11m48.555920883s) [noise=true , address=77.90.31.17:4772 , piece_id=MGZPHOODRIOTGABQJHNIP5FGF4XN6A723ZHUBKI6KNAJDKFWWOJA, seg_pos={0 0}, upload=18998]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: write tcp 188.166.225.170:42704->77.90.31.17:4772: use of closed network connection; write tcp 188.166.225.170:42704->77.90.31.17:4772: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42704->77.90.31.17:4772: use of closed network connection Aug 9 11:37:54.124845251 (18h11m48.555925382s) [noise=true , address=45.91.116.8:4047 , piece_id=B27B2QKZXICP4DHTDTCYSZZ7RN2PPCMKSRQBKRK5JJ4BE4RNWQAQ, seg_pos={0 0}, upload=18998]: Done uploading piece. err: ecclient: upload failed (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN, address:45.91.116.8:4047): protocol: write tcp 188.166.225.170:40794->45.91.116.8:4047: use of closed network connection; write tcp 188.166.225.170:40794->45.91.116.8:4047: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40794->45.91.116.8:4047: use of closed network connection Aug 9 11:37:54.124866321 (18h11m48.555946492s) [seg_pos={0 0}, upload=18998]: Done waiting for segment. successful: 6 optimal: 13 errs: <nil> Aug 9 11:37:54.353990811 (18h11m48.785070942s) [upload=18998]: Done uploading segments... Aug 9 11:37:54.354028673 (18h11m48.785108804s) [upload=18998]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 6) Duration: 5.737552635s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 3 out of 60 is failed (threshold=3) [............................X..................X.........X..] duration: 2.054469966s FAIL (2025-08-09 12:01:36 UTC) orbiter warning: operation: ListObjects ======ERROR====== unable to find key in bucket: sj://uptime-checks-placement-20/orbiter_us_0/1K ======LOGS====== Aug 9 11:59:31.819436107 (18h33m26.250516238s) [upload=19388]: Starting upload Aug 9 11:59:31.819492355 (18h33m26.250572516s) [upload=19388]: Done starting upload Aug 9 11:59:31.819563080 (18h33m26.250643211s) [upload=19388]: Uploading segments... Aug 9 11:59:31.819616853 (18h33m26.250696984s) [upload=19388]: Got next segment. Inline: true Aug 9 11:59:31.819621552 (18h33m26.250701683s) [upload=19388]: Waiting for error group managing segments... Aug 9 11:59:31.819660466 (18h33m26.250740597s) [upload=19388]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 9 11:59:31.819665206 (18h33m26.250745327s) [upload=19388]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 9 11:59:31.819667851 (18h33m26.250747972s) [upload=19388]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 9 11:59:32.242597809 (18h33m26.67367794s) [upload=19388]: Done uploading segments... Aug 9 11:59:32.242615142 (18h33m26.673695263s) [upload=19388]: Upload finished. err: <nil> Duration:* 2.054496325s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 3 out of 60 is failed (threshold=3) [...........................................X............X.X.] duration: 5.763663222s FAIL (2025-08-09 12:01:24 UTC) orbiter warning: operation: ListObjects ======ERROR====== unable to find key in bucket: sj://uptime-checks-placement-20/orbiter_us_0/4M ======LOGS====== Aug 9 12:00:23.108483377 (18h34m17.539563508s) [upload=19402]: Starting upload Aug 9 12:00:23.108636078 (18h34m17.53971621s) [upload=19402]: Done starting upload Aug 9 12:00:23.108746840 (18h34m17.539826981s) [upload=19402]: Uploading segments... Aug 9 12:00:23.108904442 (18h34m17.539984573s) [upload=19402]: Got next segment. Inline: false Aug 9 12:00:23.108970418 (18h34m17.540050599s) [upload=19402]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 9 12:00:23.108977743 (18h34m17.540057904s) [upload=19402]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 9 12:00:23.579075368 (18h34m18.010155499s) [seg_pos={0 0}, upload=19402]: Begin upload segment... Aug 9 12:00:23.579127947 (18h34m18.010208078s) [seg_pos={0 0}, upload=19402]: Done begin upload segment. Aug 9 12:00:23.579157074 (18h34m18.010237205s) [noise=true , address=104.36.82.44:4899 , piece_id=KIQ5AFZ2FIUJ3XIG5ZQBJYITK5CD4ZDW5SYOUPFIYAOZBPENWFFA, seg_pos={0 0}, upload=19402]: Uploading piece... Aug 9 12:00:23.579206939 (18h34m18.01028707s) [noise=true , address=104.36.82.55:4199 , piece_id=6D4WT6KDBSMMCPXONV74YG4IXFEXPXU2FQTCL2W2NPZ2ERVMBUMQ, seg_pos={0 0}, upload=19402]: Uploading piece... Aug 9 12:00:23.579552951 (18h34m18.010633082s) [noise=true , address=64.49.13.139:4469 , piece_id=UTZT7WZ3LQ6LOZ3C6IXV5M7L3APATJOYDRZAMWUEYCZP4I34THCA, seg_pos={0 0}, upload=19402]: Uploading piece... Aug 9 12:00:23.579921666 (18h34m18.011001797s) [noise=true , address=104.36.82.59:4327 , piece_id=WJJFOV44IRLH2E6BFC46ALBJXD54ML4HQOFSUDFNJ3OJUXASSTZQ, seg_pos={0 0}, upload=19402]: Uploading piece... Aug 9 12:00:23.580275103 (18h34m18.011355234s) [noise=true , address=104.36.82.13:4197 , piece_id=NEMRRYQPHXM73GSZVEETU7UNMUDQVNKKSLKQTRSBHBKPQ35KLLPQ, seg_pos={0 0}, upload=19402]: Uploading piece... Aug 9 12:00:23.580699144 (18h34m18.011779275s) [noise=true , address=104.36.82.23:4454 , piece_id=VNKC653W6NNZTTH6CRDK2LSD5XFLLGJGYOOIELABDCKC364DITDA, seg_pos={0 0}, upload=19402]: Uploading piece... Aug 9 12:00:23.580803934 (18h34m18.011884065s) [noise=true , address=104.36.82.61:4396 , piece_id=LJ73GMLI4ZDNLDSGIRT4GRBU56YFUOBTOC7OW3NS4TRWSUXMQCHA, seg_pos={0 0}, upload=19402]: Uploading piece... Aug 9 12:00:23.581071286 (18h34m18.012151417s) [noise=true , address=104.36.82.45:4934 , piece_id=K7XWI37JOPKYII6BFSO6AZT73S3FYOH5JKCJXOM5O5SCWAG62OIQ, seg_pos={0 0}, upload=19402]: Uploading piece... Aug 9 12:00:23.581305595 (18h34m18.012385726s) [noise=true , address=104.36.82.60:4354 , piece_id=HR52C2IWZNCL2XI553EUKTS37R5N4ILVW34GJSPWJV3Z3W7KKGWA, seg_pos={0 0}, upload=19402]: Uploading piece... Aug 9 12:00:23.581541305 (18h34m18.012621436s) [noise=true , address=45.91.116.12:4243 , piece_id=FZW53SVI3PFJ3M6WQE74FPB737JEBGBGA2YB53XCU5DEPEUQHU7Q, seg_pos={0 0}, upload=19402]: Uploading piece... Aug 9 12:00:23.581620086 (18h34m18.012700217s) [noise=true , address=104.36.82.58:4268 , piece_id=QLMKG6PZ4XYJJUTOHFW7EWTFGTST5HYLTHP5MMX6X3AJJJAZSZ3Q, seg_pos={0 0}, upload=19402]: Uploading piece... Aug 9 12:00:23.581922384 (18h34m18.013002515s) [noise=true , address=104.36.82.9:4031 , piece_id=LLSOV26QMVJRDNGWSOKPT4MIGAMTHU6AIADGGJNG6526XAR3SF6Q, seg_pos={0 0}, upload=19402]: Uploading piece... Aug 9 12:00:23.582143177 (18h34m18.013223308s) [noise=true , address=104.36.82.60:4358 , piece_id=WPW7FORNXWY6DOAXVH775D4W3VFOH7PRMQO5UXKJNIFFCRVSE43A, seg_pos={0 0}, upload=19402]: Uploading piece... Aug 9 12:00:23.582347538 (18h34m18.013427669s) [noise=true , address=104.36.82.14:4242 , piece_id=2JREAT5B4ZOSHBJPMPKVQOW2VDMLQIDURO7KTYRPFJAM6U5K57WQ, seg_pos={0 0}, upload=19402]: Uploading piece... Aug 9 12:00:23.582596004 (18h34m18.013676135s) [noise=true , address=64.49.13.170:4726 , piece_id=A2OWN4ZBVO3MZKLRAHERFMYAZDSLB4HMYESALCEVEH66LGUZEQOQ, seg_pos={0 0}, upload=19402]: Uploading piece... Aug 9 12:00:23.582774395 (18h34m18.013854526s) [noise=true , address=104.36.82.46:4484 , piece_id=FQHBXDSLXK2GIOSY4V5QBBPLOH7ZD3KHJVXR5IFYPMICQCWH7UFA, seg_pos={0 0}, upload=19402]: Uploading piece... Aug 9 12:00:23.583063899 (18h34m18.01414403s) [noise=true , address=77.90.31.12:4604 , piece_id=HPMFA765VZOJGNXPWE5SCA4AAK7QB4HFGUSYFQWOFLVZTU6PSUMA, seg_pos={0 0}, upload=19402]: Uploading piece... Aug 9 12:00:23.583138481 (18h34m18.014218612s) [noise=true , address=104.36.82.5:4845 , piece_id=5AJNZJD6MFKE3U6EZ27IEXQEO4KOUGSXA7EBCDSWVTVFYD4AJTHQ, seg_pos={0 0}, upload=19402]: Uploading piece... Aug 9 12:00:24.683477824 (18h34m19.114557955s) [upload=19402]: Next returned nil segment Aug 9 12:00:24.683489506 (18h34m19.114569637s) [upload=19402]: Waiting for error group managing segments... Aug 9 12:00:24.955322603 (18h34m19.386402734s) [noise=true , address=64.49.13.170:4726 , piece_id=A2OWN4ZBVO3MZKLRAHERFMYAZDSLB4HMYESALCEVEH66LGUZEQOQ, seg_pos={0 0}, upload=19402]: Done uploading piece. err: <nil> Aug 9 12:00:24.969832404 (18h34m19.400912535s) [noise=true , address=64.49.13.139:4469 , piece_id=UTZT7WZ3LQ6LOZ3C6IXV5M7L3APATJOYDRZAMWUEYCZP4I34THCA, seg_pos={0 0}, upload=19402]: Done uploading piece. err: <nil> Aug 9 12:00:25.516280300 (18h34m19.947360431s) [noise=true , address=104.36.82.55:4199 , piece_id=6D4WT6KDBSMMCPXONV74YG4IXFEXPXU2FQTCL2W2NPZ2ERVMBUMQ, seg_pos={0 0}, upload=19402]: Done uploading piece. err: <nil> Aug 9 12:00:25.525386162 (18h34m19.956466293s) [noise=true , address=104.36.82.46:4484 , piece_id=FQHBXDSLXK2GIOSY4V5QBBPLOH7ZD3KHJVXR5IFYPMICQCWH7UFA, seg_pos={0 0}, upload=19402]: Done uploading piece. err: <nil> Aug 9 12:00:25.526709204 (18h34m19.957789335s) [noise=true , address=104.36.82.60:4358 , piece_id=WPW7FORNXWY6DOAXVH775D4W3VFOH7PRMQO5UXKJNIFFCRVSE43A, seg_pos={0 0}, upload=19402]: Done uploading piece. err: <nil> Aug 9 12:00:25.530790893 (18h34m19.961871024s) [noise=true , address=104.36.82.44:4899 , piece_id=KIQ5AFZ2FIUJ3XIG5ZQBJYITK5CD4ZDW5SYOUPFIYAOZBPENWFFA, seg_pos={0 0}, upload=19402]: Done uploading piece. err: <nil> Aug 9 12:00:25.531900687 (18h34m19.962980818s) [noise=true , address=104.36.82.58:4268 , piece_id=QLMKG6PZ4XYJJUTOHFW7EWTFGTST5HYLTHP5MMX6X3AJJJAZSZ3Q, seg_pos={0 0}, upload=19402]: Done uploading piece. err: <nil> Aug 9 12:00:25.536797967 (18h34m19.967878098s) [noise=true , address=104.36.82.9:4031 , piece_id=LLSOV26QMVJRDNGWSOKPT4MIGAMTHU6AIADGGJNG6526XAR3SF6Q, seg_pos={0 0}, upload=19402]: Done uploading piece. err: <nil> Aug 9 12:00:25.545685641 (18h34m19.976765772s) [noise=true , address=104.36.82.60:4354 , piece_id=HR52C2IWZNCL2XI553EUKTS37R5N4ILVW34GJSPWJV3Z3W7KKGWA, seg_pos={0 0}, upload=19402]: Done uploading piece. err: <nil> Aug 9 12:00:25.555951362 (18h34m19.987031493s) [noise=true , address=104.36.82.61:4396 , piece_id=LJ73GMLI4ZDNLDSGIRT4GRBU56YFUOBTOC7OW3NS4TRWSUXMQCHA, seg_pos={0 0}, upload=19402]: Done uploading piece. err: <nil> Aug 9 12:00:25.566816712 (18h34m19.997896843s) [noise=true , address=104.36.82.5:4845 , piece_id=5AJNZJD6MFKE3U6EZ27IEXQEO4KOUGSXA7EBCDSWVTVFYD4AJTHQ, seg_pos={0 0}, upload=19402]: Done uploading piece. err: <nil> Aug 9 12:00:25.585861129 (18h34m20.01694126s) [noise=true , address=104.36.82.45:4934 , piece_id=K7XWI37JOPKYII6BFSO6AZT73S3FYOH5JKCJXOM5O5SCWAG62OIQ, seg_pos={0 0}, upload=19402]: Done uploading piece. err: <nil> Aug 9 12:00:25.594196727 (18h34m20.025276858s) [noise=true , address=45.91.116.12:4243 , piece_id=FZW53SVI3PFJ3M6WQE74FPB737JEBGBGA2YB53XCU5DEPEUQHU7Q, seg_pos={0 0}, upload=19402]: Done uploading piece. err: <nil> Aug 9 12:00:25.606123457 (18h34m20.037203588s) [noise=true , address=77.90.31.12:4604 , piece_id=HPMFA765VZOJGNXPWE5SCA4AAK7QB4HFGUSYFQWOFLVZTU6PSUMA, seg_pos={0 0}, upload=19402]: Done uploading piece. err: <nil> Aug 9 12:00:25.606140419 (18h34m20.03722055s) [seg_pos={0 0}, upload=19402]: Segment reached optimal threshold of 14 pieces. Aug 9 12:00:25.606241462 (18h34m20.037321593s) [noise=true , address=104.36.82.13:4197 , piece_id=NEMRRYQPHXM73GSZVEETU7UNMUDQVNKKSLKQTRSBHBKPQ35KLLPQ, seg_pos={0 0}, upload=19402]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1cB89C5xnya9WGD9dGBsSouqkZH3nLqytrg9Qo4UH2kdFonTjM): protocol: expected piece hash; context canceled; EOF Aug 9 12:00:25.606346553 (18h34m20.037426675s) [noise=true , address=104.36.82.14:4242 , piece_id=2JREAT5B4ZOSHBJPMPKVQOW2VDMLQIDURO7KTYRPFJAM6U5K57WQ, seg_pos={0 0}, upload=19402]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1inR8rjUp4fRQWcaD2S7MRxveHyFxKoutWoEiuKQG3C895PaMz): protocol: expected piece hash; context canceled; EOF Aug 9 12:00:25.606405596 (18h34m20.037485727s) [noise=true , address=104.36.82.23:4454 , piece_id=VNKC653W6NNZTTH6CRDK2LSD5XFLLGJGYOOIELABDCKC364DITDA, seg_pos={0 0}, upload=19402]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1UCkuYGdVYLoR8oc1jpCZCN2smNfVqFwC8a7q95fJb8haB3MY2): protocol: expected piece hash; context canceled; EOF Aug 9 12:00:25.606463838 (18h34m20.037543969s) [noise=true , address=104.36.82.59:4327 , piece_id=WJJFOV44IRLH2E6BFC46ALBJXD54ML4HQOFSUDFNJ3OJUXASSTZQ, seg_pos={0 0}, upload=19402]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1V4C6AWGbyYQWfinoSj7iKovHeTJ9hqypyC5U9rZ8pveXBBfCY): protocol: expected piece hash; context canceled; EOF Aug 9 12:00:25.606478186 (18h34m20.037558317s) [seg_pos={0 0}, upload=19402]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 9 12:00:25.606535966 (18h34m20.037616097s) [upload=19402]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 9 12:00:25.606540855 (18h34m20.037620976s) [upload=19402]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 9 12:00:25.913625167 (18h34m20.344705298s) [upload=19402]: Done uploading segments... Aug 9 12:00:25.913644565 (18h34m20.344724696s) [upload=19402]: Upload finished. err: <nil> Duration: 5.763692679s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 3 out of 60 is failed (threshold=3) [.............................X..................X.........X.] duration: 1.858935912s FAIL (2025-08-09 12:00:41 UTC) orbiter warning: operation: ListObjects ======ERROR====== unable to find key in bucket: sj://uptime-checks-placement-20/orbiter_us_0/1K ======LOGS====== Aug 9 11:59:31.819436107 (18h33m26.250516238s) [upload=19388]: Starting upload Aug 9 11:59:31.819492355 (18h33m26.250572516s) [upload=19388]: Done starting upload Aug 9 11:59:31.819563080 (18h33m26.250643211s) [upload=19388]: Uploading segments... Aug 9 11:59:31.819616853 (18h33m26.250696984s) [upload=19388]: Got next segment. Inline: true Aug 9 11:59:31.819621552 (18h33m26.250701683s) [upload=19388]: Waiting for error group managing segments... Aug 9 11:59:31.819660466 (18h33m26.250740597s) [upload=19388]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 9 11:59:31.819665206 (18h33m26.250745327s) [upload=19388]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 9 11:59:31.819667851 (18h33m26.250747972s) [upload=19388]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 9 11:59:32.242597809 (18h33m26.67367794s) [upload=19388]: Done uploading segments... Aug 9 11:59:32.242615142 (18h33m26.673695263s) [upload=19388]: Upload finished. err: <nil> Duration:* 1.858963345s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................X..............................] duration: 5.80884289s FAIL (2025-08-09 12:00:40 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 6) ======LOGS====== ADHZNIB6LBP5VQHFEGSYM3YNNZWNM2N4UU7JTCQ, seg_pos={0 0}, upload=18998]: Uploading piece... Aug 9 11:37:25.005421531 (18h11m19.436501662s) [noise=true , address=45.91.116.4:4940 , piece_id=M33ZGUCEF23HDGQZE5F5KLWQ43NHC7E7LKOVVGQ6P6TM2HFGB4WQ, seg_pos={0 0}, upload=18998]: Uploading piece... Aug 9 11:37:25.005614329 (18h11m19.43669446s) [noise=true , address=45.91.116.8:4047 , piece_id=B27B2QKZXICP4DHTDTCYSZZ7RN2PPCMKSRQBKRK5JJ4BE4RNWQAQ, seg_pos={0 0}, upload=18998]: Uploading piece... Aug 9 11:37:25.005727536 (18h11m19.436807667s) [noise=true , address=45.91.116.12:4225 , piece_id=VFK2SMVTBT76D7YLQWGRA6TF47JD3B7J7F5NFR4E2MQVNGNHYTAQ, seg_pos={0 0}, upload=18998]: Uploading piece... Aug 9 11:37:25.006281517 (18h11m19.437361648s) [noise=true , address=104.36.82.7:4909 , piece_id=OGG5ZCXSLIM5YKMM7K6KLKLWGOTX376DUTDT3567OB3AJIBPBKDQ, seg_pos={0 0}, upload=18998]: Uploading piece... Aug 9 11:37:25.006632558 (18h11m19.437712679s) [noise=true , address=45.91.116.13:4155 , piece_id=K4CGD4OCR2B6ZWAZHB5MDBECVJB3AC2HDXS2TUVAVGIWQYJEARLA, seg_pos={0 0}, upload=18998]: Uploading piece... Aug 9 11:37:25.006996264 (18h11m19.438076395s) [noise=true , address=77.90.31.18:4806 , piece_id=QCURQYEJL22LBT2YFBR62RIHH7VNSGL7GERP66GKHXK3JP5WLVZQ, seg_pos={0 0}, upload=18998]: Uploading piece... Aug 9 11:37:25.007245231 (18h11m19.438325362s) [noise=true , address=104.36.82.3:4765 , piece_id=HBLTTPZYK5ZXNG2XV53QVJA3NPX3YWKEAAAFVMDXUO6BLR3QKCWQ, seg_pos={0 0}, upload=18998]: Uploading piece... Aug 9 11:37:25.007475481 (18h11m19.438555612s) [noise=true , address=45.91.116.6:4973 , piece_id=MHDNXTVKFMLRKGS7JIPL4U5VQMBXGP4EYM7JEGPSK4U3GIJ5JEHQ, seg_pos={0 0}, upload=18998]: Uploading piece... Aug 9 11:37:25.007873984 (18h11m19.438954115s) [noise=true , address=45.91.116.7:4010 , piece_id=5G3O552GD5XO7SNIHST623QJBWJF2C7B6EN7ISGHHY3FYV6QL2DQ, seg_pos={0 0}, upload=18998]: Uploading piece... Aug 9 11:37:26.733124951 (18h11m21.164205082s) [upload=18998]: Next returned nil segment Aug 9 11:37:26.733142194 (18h11m21.164222325s) [upload=18998]: Waiting for error group managing segments... Aug 9 11:37:27.100757697 (18h11m21.531837848s) [noise=true , address=104.36.82.14:4222 , piece_id=Z4X6RJ5HBAYGUNKT6DXWSS2TEG2OUFPMH3NWYGA7D26XIFH46UAA, seg_pos={0 0}, upload=18998]: Done uploading piece. err: <nil> Aug 9 11:37:27.106781913 (18h11m21.537862064s) [noise=true , address=104.36.82.7:4912 , piece_id=MRA5X6N6MDQ2TADHZNIB6LBP5VQHFEGSYM3YNNZWNM2N4UU7JTCQ, seg_pos={0 0}, upload=18998]: Done uploading piece. err: <nil> Aug 9 11:37:27.136679105 (18h11m21.567759236s) [noise=true , address=104.36.82.5:4838 , piece_id=T3WCWCLB4WAXQY3S6VCFHLKYK6IY2Q6P2776CLGYS4QOFTXMYVEQ, seg_pos={0 0}, upload=18998]: Done uploading piece. err: <nil> Aug 9 11:37:27.139126447 (18h11m21.570206578s) [noise=true , address=104.36.82.3:4765 , piece_id=HBLTTPZYK5ZXNG2XV53QVJA3NPX3YWKEAAAFVMDXUO6BLR3QKCWQ, seg_pos={0 0}, upload=18998]: Done uploading piece. err: <nil> Aug 9 11:37:27.183514474 (18h11m21.614594606s) [noise=true , address=104.36.82.7:4909 , piece_id=OGG5ZCXSLIM5YKMM7K6KLKLWGOTX376DUTDT3567OB3AJIBPBKDQ, seg_pos={0 0}, upload=18998]: Done uploading piece. err: <nil> Aug 9 11:37:53.489259032 (18h11m47.920339163s) [noise=true , address=45.91.116.13:4155 , piece_id=K4CGD4OCR2B6ZWAZHB5MDBECVJB3AC2HDXS2TUVAVGIWQYJEARLA, seg_pos={0 0}, upload=18998]: Done uploading piece. err: <nil> Aug 9 11:37:54.124525138 (18h11m48.555605259s) [noise=true , address=45.91.116.7:4010 , piece_id=5G3O552GD5XO7SNIHST623QJBWJF2C7B6EN7ISGHHY3FYV6QL2DQ, seg_pos={0 0}, upload=18998]: Done uploading piece. err: ecclient: upload failed (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w, address:45.91.116.7:4010): protocol: write tcp 188.166.225.170:54960->45.91.116.7:4010: use of closed network connection; write tcp 188.166.225.170:54960->45.91.116.7:4010: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54960->45.91.116.7:4010: use of closed network connection Aug 9 11:37:54.124581035 (18h11m48.555661166s) [noise=true , address=77.90.31.10:4520 , piece_id=YRBBWM54FGU3B74JQ7EOAO7VSGVFXAVT4SNZ6KN3IQFIHIEVV5HA, seg_pos={0 0}, upload=18998]: Done uploading piece. err: ecclient: upload failed (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp, address:77.90.31.10:4520): protocol: write tcp 188.166.225.170:39560->77.90.31.10:4520: use of closed network connection; write tcp 188.166.225.170:39560->77.90.31.10:4520: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39560->77.90.31.10:4520: use of closed network connection Aug 9 11:37:54.124615480 (18h11m48.555695652s) [noise=true , address=45.91.116.12:4225 , piece_id=VFK2SMVTBT76D7YLQWGRA6TF47JD3B7J7F5NFR4E2MQVNGNHYTAQ, seg_pos={0 0}, upload=18998]: Done uploading piece. err: ecclient: upload failed (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy, address:45.91.116.12:4225): protocol: write tcp 188.166.225.170:38054->45.91.116.12:4225: use of closed network connection; write tcp 188.166.225.170:38054->45.91.116.12:4225: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38054->45.91.116.12:4225: use of closed network connection Aug 9 11:37:54.124643875 (18h11m48.555724016s) [noise=true , address=45.91.116.4:4940 , piece_id=M33ZGUCEF23HDGQZE5F5KLWQ43NHC7E7LKOVVGQ6P6TM2HFGB4WQ, seg_pos={0 0}, upload=18998]: Done uploading piece. err: ecclient: upload failed (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe, address:45.91.116.4:4940): protocol: write tcp 188.166.225.170:40630->45.91.116.4:4940: use of closed network connection; write tcp 188.166.225.170:40630->45.91.116.4:4940: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40630->45.91.116.4:4940: use of closed network connection Aug 9 11:37:54.124693159 (18h11m48.55577335s) [noise=true , address=45.91.116.14:4189 , piece_id=XVQMTYHB6H3IKSERYT2BEB4NP42KV6FRE6J2MKKU5WEOCVBHYIWA, seg_pos={0 0}, upload=18998]: Done uploading piece. err: ecclient: upload failed (node:1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N, address:45.91.116.14:4189): protocol: write tcp 188.166.225.170:37650->45.91.116.14:4189: use of closed network connection; write tcp 188.166.225.170:37650->45.91.116.14:4189: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37650->45.91.116.14:4189: use of closed network connection Aug 9 11:37:54.124771740 (18h11m48.555851871s) [noise=true , address=77.90.31.18:4806 , piece_id=QCURQYEJL22LBT2YFBR62RIHH7VNSGL7GERP66GKHXK3JP5WLVZQ, seg_pos={0 0}, upload=18998]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: write tcp 188.166.225.170:55536->77.90.31.18:4806: use of closed network connection; write tcp 188.166.225.170:55536->77.90.31.18:4806: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55536->77.90.31.18:4806: use of closed network connection Aug 9 11:37:54.124817217 (18h11m48.555897348s) [noise=true , address=77.90.31.15:4697 , piece_id=Q26SRGZO2YBJWPDIG4U422EICE6XEXPLHFHXLKFUBKMMJAZ4W7UQ, seg_pos={0 0}, upload=18998]: Done uploading piece. err: ecclient: upload failed (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, address:77.90.31.15:4697): protocol: write tcp 188.166.225.170:49618->77.90.31.15:4697: use of closed network connection; write tcp 188.166.225.170:49618->77.90.31.15:4697: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49618->77.90.31.15:4697: use of closed network connection Aug 9 11:37:54.124833387 (18h11m48.555913559s) [noise=true , address=45.91.116.6:4973 , piece_id=MHDNXTVKFMLRKGS7JIPL4U5VQMBXGP4EYM7JEGPSK4U3GIJ5JEHQ, seg_pos={0 0}, upload=18998]: Done uploading piece. err: ecclient: upload failed (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, address:45.91.116.6:4973): protocol: write tcp 188.166.225.170:59438->45.91.116.6:4973: use of closed network connection; write tcp 188.166.225.170:59438->45.91.116.6:4973: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59438->45.91.116.6:4973: use of closed network connection Aug 9 11:37:54.124840752 (18h11m48.555920883s) [noise=true , address=77.90.31.17:4772 , piece_id=MGZPHOODRIOTGABQJHNIP5FGF4XN6A723ZHUBKI6KNAJDKFWWOJA, seg_pos={0 0}, upload=18998]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: write tcp 188.166.225.170:42704->77.90.31.17:4772: use of closed network connection; write tcp 188.166.225.170:42704->77.90.31.17:4772: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42704->77.90.31.17:4772: use of closed network connection Aug 9 11:37:54.124845251 (18h11m48.555925382s) [noise=true , address=45.91.116.8:4047 , piece_id=B27B2QKZXICP4DHTDTCYSZZ7RN2PPCMKSRQBKRK5JJ4BE4RNWQAQ, seg_pos={0 0}, upload=18998]: Done uploading piece. err: ecclient: upload failed (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN, address:45.91.116.8:4047): protocol: write tcp 188.166.225.170:40794->45.91.116.8:4047: use of closed network connection; write tcp 188.166.225.170:40794->45.91.116.8:4047: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40794->45.91.116.8:4047: use of closed network connection Aug 9 11:37:54.124866321 (18h11m48.555946492s) [seg_pos={0 0}, upload=18998]: Done waiting for segment. successful: 6 optimal: 13 errs: <nil> Aug 9 11:37:54.353990811 (18h11m48.785070942s) [upload=18998]: Done uploading segments... Aug 9 11:37:54.354028673 (18h11m48.785108804s) [upload=18998]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 6) Duration: 5.808871717s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 3 out of 60 is failed (threshold=3) [............................................X............X.X] duration: 3.903324668s FAIL (2025-08-09 12:00:26 UTC) orbiter warning: operation: ListObjects ======ERROR====== unable to find key in bucket: sj://uptime-checks-placement-20/orbiter_us_0/4M ======LOGS====== Aug 9 12:00:23.108483377 (18h34m17.539563508s) [upload=19402]: Starting upload Aug 9 12:00:23.108636078 (18h34m17.53971621s) [upload=19402]: Done starting upload Aug 9 12:00:23.108746840 (18h34m17.539826981s) [upload=19402]: Uploading segments... Aug 9 12:00:23.108904442 (18h34m17.539984573s) [upload=19402]: Got next segment. Inline: false Aug 9 12:00:23.108970418 (18h34m17.540050599s) [upload=19402]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 9 12:00:23.108977743 (18h34m17.540057904s) [upload=19402]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 9 12:00:23.579075368 (18h34m18.010155499s) [seg_pos={0 0}, upload=19402]: Begin upload segment... Aug 9 12:00:23.579127947 (18h34m18.010208078s) [seg_pos={0 0}, upload=19402]: Done begin upload segment. Aug 9 12:00:23.579157074 (18h34m18.010237205s) [noise=true , address=104.36.82.44:4899 , piece_id=KIQ5AFZ2FIUJ3XIG5ZQBJYITK5CD4ZDW5SYOUPFIYAOZBPENWFFA, seg_pos={0 0}, upload=19402]: Uploading piece... Aug 9 12:00:23.579206939 (18h34m18.01028707s) [noise=true , address=104.36.82.55:4199 , piece_id=6D4WT6KDBSMMCPXONV74YG4IXFEXPXU2FQTCL2W2NPZ2ERVMBUMQ, seg_pos={0 0}, upload=19402]: Uploading piece... Aug 9 12:00:23.579552951 (18h34m18.010633082s) [noise=true , address=64.49.13.139:4469 , piece_id=UTZT7WZ3LQ6LOZ3C6IXV5M7L3APATJOYDRZAMWUEYCZP4I34THCA, seg_pos={0 0}, upload=19402]: Uploading piece... Aug 9 12:00:23.579921666 (18h34m18.011001797s) [noise=true , address=104.36.82.59:4327 , piece_id=WJJFOV44IRLH2E6BFC46ALBJXD54ML4HQOFSUDFNJ3OJUXASSTZQ, seg_pos={0 0}, upload=19402]: Uploading piece... Aug 9 12:00:23.580275103 (18h34m18.011355234s) [noise=true , address=104.36.82.13:4197 , piece_id=NEMRRYQPHXM73GSZVEETU7UNMUDQVNKKSLKQTRSBHBKPQ35KLLPQ, seg_pos={0 0}, upload=19402]: Uploading piece... Aug 9 12:00:23.580699144 (18h34m18.011779275s) [noise=true , address=104.36.82.23:4454 , piece_id=VNKC653W6NNZTTH6CRDK2LSD5XFLLGJGYOOIELABDCKC364DITDA, seg_pos={0 0}, upload=19402]: Uploading piece... Aug 9 12:00:23.580803934 (18h34m18.011884065s) [noise=true , address=104.36.82.61:4396 , piece_id=LJ73GMLI4ZDNLDSGIRT4GRBU56YFUOBTOC7OW3NS4TRWSUXMQCHA, seg_pos={0 0}, upload=19402]: Uploading piece... Aug 9 12:00:23.581071286 (18h34m18.012151417s) [noise=true , address=104.36.82.45:4934 , piece_id=K7XWI37JOPKYII6BFSO6AZT73S3FYOH5JKCJXOM5O5SCWAG62OIQ, seg_pos={0 0}, upload=19402]: Uploading piece... Aug 9 12:00:23.581305595 (18h34m18.012385726s) [noise=true , address=104.36.82.60:4354 , piece_id=HR52C2IWZNCL2XI553EUKTS37R5N4ILVW34GJSPWJV3Z3W7KKGWA, seg_pos={0 0}, upload=19402]: Uploading piece... Aug 9 12:00:23.581541305 (18h34m18.012621436s) [noise=true , address=45.91.116.12:4243 , piece_id=FZW53SVI3PFJ3M6WQE74FPB737JEBGBGA2YB53XCU5DEPEUQHU7Q, seg_pos={0 0}, upload=19402]: Uploading piece... Aug 9 12:00:23.581620086 (18h34m18.012700217s) [noise=true , address=104.36.82.58:4268 , piece_id=QLMKG6PZ4XYJJUTOHFW7EWTFGTST5HYLTHP5MMX6X3AJJJAZSZ3Q, seg_pos={0 0}, upload=19402]: Uploading piece... Aug 9 12:00:23.581922384 (18h34m18.013002515s) [noise=true , address=104.36.82.9:4031 , piece_id=LLSOV26QMVJRDNGWSOKPT4MIGAMTHU6AIADGGJNG6526XAR3SF6Q, seg_pos={0 0}, upload=19402]: Uploading piece... Aug 9 12:00:23.582143177 (18h34m18.013223308s) [noise=true , address=104.36.82.60:4358 , piece_id=WPW7FORNXWY6DOAXVH775D4W3VFOH7PRMQO5UXKJNIFFCRVSE43A, seg_pos={0 0}, upload=19402]: Uploading piece... Aug 9 12:00:23.582347538 (18h34m18.013427669s) [noise=true , address=104.36.82.14:4242 , piece_id=2JREAT5B4ZOSHBJPMPKVQOW2VDMLQIDURO7KTYRPFJAM6U5K57WQ, seg_pos={0 0}, upload=19402]: Uploading piece... Aug 9 12:00:23.582596004 (18h34m18.013676135s) [noise=true , address=64.49.13.170:4726 , piece_id=A2OWN4ZBVO3MZKLRAHERFMYAZDSLB4HMYESALCEVEH66LGUZEQOQ, seg_pos={0 0}, upload=19402]: Uploading piece... Aug 9 12:00:23.582774395 (18h34m18.013854526s) [noise=true , address=104.36.82.46:4484 , piece_id=FQHBXDSLXK2GIOSY4V5QBBPLOH7ZD3KHJVXR5IFYPMICQCWH7UFA, seg_pos={0 0}, upload=19402]: Uploading piece... Aug 9 12:00:23.583063899 (18h34m18.01414403s) [noise=true , address=77.90.31.12:4604 , piece_id=HPMFA765VZOJGNXPWE5SCA4AAK7QB4HFGUSYFQWOFLVZTU6PSUMA, seg_pos={0 0}, upload=19402]: Uploading piece... Aug 9 12:00:23.583138481 (18h34m18.014218612s) [noise=true , address=104.36.82.5:4845 , piece_id=5AJNZJD6MFKE3U6EZ27IEXQEO4KOUGSXA7EBCDSWVTVFYD4AJTHQ, seg_pos={0 0}, upload=19402]: Uploading piece... Aug 9 12:00:24.683477824 (18h34m19.114557955s) [upload=19402]: Next returned nil segment Aug 9 12:00:24.683489506 (18h34m19.114569637s) [upload=19402]: Waiting for error group managing segments... Aug 9 12:00:24.955322603 (18h34m19.386402734s) [noise=true , address=64.49.13.170:4726 , piece_id=A2OWN4ZBVO3MZKLRAHERFMYAZDSLB4HMYESALCEVEH66LGUZEQOQ, seg_pos={0 0}, upload=19402]: Done uploading piece. err: <nil> Aug 9 12:00:24.969832404 (18h34m19.400912535s) [noise=true , address=64.49.13.139:4469 , piece_id=UTZT7WZ3LQ6LOZ3C6IXV5M7L3APATJOYDRZAMWUEYCZP4I34THCA, seg_pos={0 0}, upload=19402]: Done uploading piece. err: <nil> Aug 9 12:00:25.516280300 (18h34m19.947360431s) [noise=true , address=104.36.82.55:4199 , piece_id=6D4WT6KDBSMMCPXONV74YG4IXFEXPXU2FQTCL2W2NPZ2ERVMBUMQ, seg_pos={0 0}, upload=19402]: Done uploading piece. err: <nil> Aug 9 12:00:25.525386162 (18h34m19.956466293s) [noise=true , address=104.36.82.46:4484 , piece_id=FQHBXDSLXK2GIOSY4V5QBBPLOH7ZD3KHJVXR5IFYPMICQCWH7UFA, seg_pos={0 0}, upload=19402]: Done uploading piece. err: <nil> Aug 9 12:00:25.526709204 (18h34m19.957789335s) [noise=true , address=104.36.82.60:4358 , piece_id=WPW7FORNXWY6DOAXVH775D4W3VFOH7PRMQO5UXKJNIFFCRVSE43A, seg_pos={0 0}, upload=19402]: Done uploading piece. err: <nil> Aug 9 12:00:25.530790893 (18h34m19.961871024s) [noise=true , address=104.36.82.44:4899 , piece_id=KIQ5AFZ2FIUJ3XIG5ZQBJYITK5CD4ZDW5SYOUPFIYAOZBPENWFFA, seg_pos={0 0}, upload=19402]: Done uploading piece. err: <nil> Aug 9 12:00:25.531900687 (18h34m19.962980818s) [noise=true , address=104.36.82.58:4268 , piece_id=QLMKG6PZ4XYJJUTOHFW7EWTFGTST5HYLTHP5MMX6X3AJJJAZSZ3Q, seg_pos={0 0}, upload=19402]: Done uploading piece. err: <nil> Aug 9 12:00:25.536797967 (18h34m19.967878098s) [noise=true , address=104.36.82.9:4031 , piece_id=LLSOV26QMVJRDNGWSOKPT4MIGAMTHU6AIADGGJNG6526XAR3SF6Q, seg_pos={0 0}, upload=19402]: Done uploading piece. err: <nil> Aug 9 12:00:25.545685641 (18h34m19.976765772s) [noise=true , address=104.36.82.60:4354 , piece_id=HR52C2IWZNCL2XI553EUKTS37R5N4ILVW34GJSPWJV3Z3W7KKGWA, seg_pos={0 0}, upload=19402]: Done uploading piece. err: <nil> Aug 9 12:00:25.555951362 (18h34m19.987031493s) [noise=true , address=104.36.82.61:4396 , piece_id=LJ73GMLI4ZDNLDSGIRT4GRBU56YFUOBTOC7OW3NS4TRWSUXMQCHA, seg_pos={0 0}, upload=19402]: Done uploading piece. err: <nil> Aug 9 12:00:25.566816712 (18h34m19.997896843s) [noise=true , address=104.36.82.5:4845 , piece_id=5AJNZJD6MFKE3U6EZ27IEXQEO4KOUGSXA7EBCDSWVTVFYD4AJTHQ, seg_pos={0 0}, upload=19402]: Done uploading piece. err: <nil> Aug 9 12:00:25.585861129 (18h34m20.01694126s) [noise=true , address=104.36.82.45:4934 , piece_id=K7XWI37JOPKYII6BFSO6AZT73S3FYOH5JKCJXOM5O5SCWAG62OIQ, seg_pos={0 0}, upload=19402]: Done uploading piece. err: <nil> Aug 9 12:00:25.594196727 (18h34m20.025276858s) [noise=true , address=45.91.116.12:4243 , piece_id=FZW53SVI3PFJ3M6WQE74FPB737JEBGBGA2YB53XCU5DEPEUQHU7Q, seg_pos={0 0}, upload=19402]: Done uploading piece. err: <nil> Aug 9 12:00:25.606123457 (18h34m20.037203588s) [noise=true , address=77.90.31.12:4604 , piece_id=HPMFA765VZOJGNXPWE5SCA4AAK7QB4HFGUSYFQWOFLVZTU6PSUMA, seg_pos={0 0}, upload=19402]: Done uploading piece. err: <nil> Aug 9 12:00:25.606140419 (18h34m20.03722055s) [seg_pos={0 0}, upload=19402]: Segment reached optimal threshold of 14 pieces. Aug 9 12:00:25.606241462 (18h34m20.037321593s) [noise=true , address=104.36.82.13:4197 , piece_id=NEMRRYQPHXM73GSZVEETU7UNMUDQVNKKSLKQTRSBHBKPQ35KLLPQ, seg_pos={0 0}, upload=19402]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1cB89C5xnya9WGD9dGBsSouqkZH3nLqytrg9Qo4UH2kdFonTjM): protocol: expected piece hash; context canceled; EOF Aug 9 12:00:25.606346553 (18h34m20.037426675s) [noise=true , address=104.36.82.14:4242 , piece_id=2JREAT5B4ZOSHBJPMPKVQOW2VDMLQIDURO7KTYRPFJAM6U5K57WQ, seg_pos={0 0}, upload=19402]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1inR8rjUp4fRQWcaD2S7MRxveHyFxKoutWoEiuKQG3C895PaMz): protocol: expected piece hash; context canceled; EOF Aug 9 12:00:25.606405596 (18h34m20.037485727s) [noise=true , address=104.36.82.23:4454 , piece_id=VNKC653W6NNZTTH6CRDK2LSD5XFLLGJGYOOIELABDCKC364DITDA, seg_pos={0 0}, upload=19402]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1UCkuYGdVYLoR8oc1jpCZCN2smNfVqFwC8a7q95fJb8haB3MY2): protocol: expected piece hash; context canceled; EOF Aug 9 12:00:25.606463838 (18h34m20.037543969s) [noise=true , address=104.36.82.59:4327 , piece_id=WJJFOV44IRLH2E6BFC46ALBJXD54ML4HQOFSUDFNJ3OJUXASSTZQ, seg_pos={0 0}, upload=19402]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1V4C6AWGbyYQWfinoSj7iKovHeTJ9hqypyC5U9rZ8pveXBBfCY): protocol: expected piece hash; context canceled; EOF Aug 9 12:00:25.606478186 (18h34m20.037558317s) [seg_pos={0 0}, upload=19402]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 9 12:00:25.606535966 (18h34m20.037616097s) [upload=19402]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 9 12:00:25.606540855 (18h34m20.037620976s) [upload=19402]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 9 12:00:25.913625167 (18h34m20.344705298s) [upload=19402]: Done uploading segments... Aug 9 12:00:25.913644565 (18h34m20.344724696s) [upload=19402]: Upload finished. err: <nil> Duration: 3.903351268s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................X.............................] duration: 5.922642095s FAIL (2025-08-09 12:00:12 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 6) ======LOGS====== ADHZNIB6LBP5VQHFEGSYM3YNNZWNM2N4UU7JTCQ, seg_pos={0 0}, upload=18998]: Uploading piece... Aug 9 11:37:25.005421531 (18h11m19.436501662s) [noise=true , address=45.91.116.4:4940 , piece_id=M33ZGUCEF23HDGQZE5F5KLWQ43NHC7E7LKOVVGQ6P6TM2HFGB4WQ, seg_pos={0 0}, upload=18998]: Uploading piece... Aug 9 11:37:25.005614329 (18h11m19.43669446s) [noise=true , address=45.91.116.8:4047 , piece_id=B27B2QKZXICP4DHTDTCYSZZ7RN2PPCMKSRQBKRK5JJ4BE4RNWQAQ, seg_pos={0 0}, upload=18998]: Uploading piece... Aug 9 11:37:25.005727536 (18h11m19.436807667s) [noise=true , address=45.91.116.12:4225 , piece_id=VFK2SMVTBT76D7YLQWGRA6TF47JD3B7J7F5NFR4E2MQVNGNHYTAQ, seg_pos={0 0}, upload=18998]: Uploading piece... Aug 9 11:37:25.006281517 (18h11m19.437361648s) [noise=true , address=104.36.82.7:4909 , piece_id=OGG5ZCXSLIM5YKMM7K6KLKLWGOTX376DUTDT3567OB3AJIBPBKDQ, seg_pos={0 0}, upload=18998]: Uploading piece... Aug 9 11:37:25.006632558 (18h11m19.437712679s) [noise=true , address=45.91.116.13:4155 , piece_id=K4CGD4OCR2B6ZWAZHB5MDBECVJB3AC2HDXS2TUVAVGIWQYJEARLA, seg_pos={0 0}, upload=18998]: Uploading piece... Aug 9 11:37:25.006996264 (18h11m19.438076395s) [noise=true , address=77.90.31.18:4806 , piece_id=QCURQYEJL22LBT2YFBR62RIHH7VNSGL7GERP66GKHXK3JP5WLVZQ, seg_pos={0 0}, upload=18998]: Uploading piece... Aug 9 11:37:25.007245231 (18h11m19.438325362s) [noise=true , address=104.36.82.3:4765 , piece_id=HBLTTPZYK5ZXNG2XV53QVJA3NPX3YWKEAAAFVMDXUO6BLR3QKCWQ, seg_pos={0 0}, upload=18998]: Uploading piece... Aug 9 11:37:25.007475481 (18h11m19.438555612s) [noise=true , address=45.91.116.6:4973 , piece_id=MHDNXTVKFMLRKGS7JIPL4U5VQMBXGP4EYM7JEGPSK4U3GIJ5JEHQ, seg_pos={0 0}, upload=18998]: Uploading piece... Aug 9 11:37:25.007873984 (18h11m19.438954115s) [noise=true , address=45.91.116.7:4010 , piece_id=5G3O552GD5XO7SNIHST623QJBWJF2C7B6EN7ISGHHY3FYV6QL2DQ, seg_pos={0 0}, upload=18998]: Uploading piece... Aug 9 11:37:26.733124951 (18h11m21.164205082s) [upload=18998]: Next returned nil segment Aug 9 11:37:26.733142194 (18h11m21.164222325s) [upload=18998]: Waiting for error group managing segments... Aug 9 11:37:27.100757697 (18h11m21.531837848s) [noise=true , address=104.36.82.14:4222 , piece_id=Z4X6RJ5HBAYGUNKT6DXWSS2TEG2OUFPMH3NWYGA7D26XIFH46UAA, seg_pos={0 0}, upload=18998]: Done uploading piece. err: <nil> Aug 9 11:37:27.106781913 (18h11m21.537862064s) [noise=true , address=104.36.82.7:4912 , piece_id=MRA5X6N6MDQ2TADHZNIB6LBP5VQHFEGSYM3YNNZWNM2N4UU7JTCQ, seg_pos={0 0}, upload=18998]: Done uploading piece. err: <nil> Aug 9 11:37:27.136679105 (18h11m21.567759236s) [noise=true , address=104.36.82.5:4838 , piece_id=T3WCWCLB4WAXQY3S6VCFHLKYK6IY2Q6P2776CLGYS4QOFTXMYVEQ, seg_pos={0 0}, upload=18998]: Done uploading piece. err: <nil> Aug 9 11:37:27.139126447 (18h11m21.570206578s) [noise=true , address=104.36.82.3:4765 , piece_id=HBLTTPZYK5ZXNG2XV53QVJA3NPX3YWKEAAAFVMDXUO6BLR3QKCWQ, seg_pos={0 0}, upload=18998]: Done uploading piece. err: <nil> Aug 9 11:37:27.183514474 (18h11m21.614594606s) [noise=true , address=104.36.82.7:4909 , piece_id=OGG5ZCXSLIM5YKMM7K6KLKLWGOTX376DUTDT3567OB3AJIBPBKDQ, seg_pos={0 0}, upload=18998]: Done uploading piece. err: <nil> Aug 9 11:37:53.489259032 (18h11m47.920339163s) [noise=true , address=45.91.116.13:4155 , piece_id=K4CGD4OCR2B6ZWAZHB5MDBECVJB3AC2HDXS2TUVAVGIWQYJEARLA, seg_pos={0 0}, upload=18998]: Done uploading piece. err: <nil> Aug 9 11:37:54.124525138 (18h11m48.555605259s) [noise=true , address=45.91.116.7:4010 , piece_id=5G3O552GD5XO7SNIHST623QJBWJF2C7B6EN7ISGHHY3FYV6QL2DQ, seg_pos={0 0}, upload=18998]: Done uploading piece. err: ecclient: upload failed (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w, address:45.91.116.7:4010): protocol: write tcp 188.166.225.170:54960->45.91.116.7:4010: use of closed network connection; write tcp 188.166.225.170:54960->45.91.116.7:4010: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54960->45.91.116.7:4010: use of closed network connection Aug 9 11:37:54.124581035 (18h11m48.555661166s) [noise=true , address=77.90.31.10:4520 , piece_id=YRBBWM54FGU3B74JQ7EOAO7VSGVFXAVT4SNZ6KN3IQFIHIEVV5HA, seg_pos={0 0}, upload=18998]: Done uploading piece. err: ecclient: upload failed (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp, address:77.90.31.10:4520): protocol: write tcp 188.166.225.170:39560->77.90.31.10:4520: use of closed network connection; write tcp 188.166.225.170:39560->77.90.31.10:4520: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39560->77.90.31.10:4520: use of closed network connection Aug 9 11:37:54.124615480 (18h11m48.555695652s) [noise=true , address=45.91.116.12:4225 , piece_id=VFK2SMVTBT76D7YLQWGRA6TF47JD3B7J7F5NFR4E2MQVNGNHYTAQ, seg_pos={0 0}, upload=18998]: Done uploading piece. err: ecclient: upload failed (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy, address:45.91.116.12:4225): protocol: write tcp 188.166.225.170:38054->45.91.116.12:4225: use of closed network connection; write tcp 188.166.225.170:38054->45.91.116.12:4225: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38054->45.91.116.12:4225: use of closed network connection Aug 9 11:37:54.124643875 (18h11m48.555724016s) [noise=true , address=45.91.116.4:4940 , piece_id=M33ZGUCEF23HDGQZE5F5KLWQ43NHC7E7LKOVVGQ6P6TM2HFGB4WQ, seg_pos={0 0}, upload=18998]: Done uploading piece. err: ecclient: upload failed (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe, address:45.91.116.4:4940): protocol: write tcp 188.166.225.170:40630->45.91.116.4:4940: use of closed network connection; write tcp 188.166.225.170:40630->45.91.116.4:4940: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40630->45.91.116.4:4940: use of closed network connection Aug 9 11:37:54.124693159 (18h11m48.55577335s) [noise=true , address=45.91.116.14:4189 , piece_id=XVQMTYHB6H3IKSERYT2BEB4NP42KV6FRE6J2MKKU5WEOCVBHYIWA, seg_pos={0 0}, upload=18998]: Done uploading piece. err: ecclient: upload failed (node:1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N, address:45.91.116.14:4189): protocol: write tcp 188.166.225.170:37650->45.91.116.14:4189: use of closed network connection; write tcp 188.166.225.170:37650->45.91.116.14:4189: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37650->45.91.116.14:4189: use of closed network connection Aug 9 11:37:54.124771740 (18h11m48.555851871s) [noise=true , address=77.90.31.18:4806 , piece_id=QCURQYEJL22LBT2YFBR62RIHH7VNSGL7GERP66GKHXK3JP5WLVZQ, seg_pos={0 0}, upload=18998]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: write tcp 188.166.225.170:55536->77.90.31.18:4806: use of closed network connection; write tcp 188.166.225.170:55536->77.90.31.18:4806: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55536->77.90.31.18:4806: use of closed network connection Aug 9 11:37:54.124817217 (18h11m48.555897348s) [noise=true , address=77.90.31.15:4697 , piece_id=Q26SRGZO2YBJWPDIG4U422EICE6XEXPLHFHXLKFUBKMMJAZ4W7UQ, seg_pos={0 0}, upload=18998]: Done uploading piece. err: ecclient: upload failed (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, address:77.90.31.15:4697): protocol: write tcp 188.166.225.170:49618->77.90.31.15:4697: use of closed network connection; write tcp 188.166.225.170:49618->77.90.31.15:4697: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49618->77.90.31.15:4697: use of closed network connection Aug 9 11:37:54.124833387 (18h11m48.555913559s) [noise=true , address=45.91.116.6:4973 , piece_id=MHDNXTVKFMLRKGS7JIPL4U5VQMBXGP4EYM7JEGPSK4U3GIJ5JEHQ, seg_pos={0 0}, upload=18998]: Done uploading piece. err: ecclient: upload failed (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, address:45.91.116.6:4973): protocol: write tcp 188.166.225.170:59438->45.91.116.6:4973: use of closed network connection; write tcp 188.166.225.170:59438->45.91.116.6:4973: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59438->45.91.116.6:4973: use of closed network connection Aug 9 11:37:54.124840752 (18h11m48.555920883s) [noise=true , address=77.90.31.17:4772 , piece_id=MGZPHOODRIOTGABQJHNIP5FGF4XN6A723ZHUBKI6KNAJDKFWWOJA, seg_pos={0 0}, upload=18998]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: write tcp 188.166.225.170:42704->77.90.31.17:4772: use of closed network connection; write tcp 188.166.225.170:42704->77.90.31.17:4772: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42704->77.90.31.17:4772: use of closed network connection Aug 9 11:37:54.124845251 (18h11m48.555925382s) [noise=true , address=45.91.116.8:4047 , piece_id=B27B2QKZXICP4DHTDTCYSZZ7RN2PPCMKSRQBKRK5JJ4BE4RNWQAQ, seg_pos={0 0}, upload=18998]: Done uploading piece. err: ecclient: upload failed (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN, address:45.91.116.8:4047): protocol: write tcp 188.166.225.170:40794->45.91.116.8:4047: use of closed network connection; write tcp 188.166.225.170:40794->45.91.116.8:4047: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40794->45.91.116.8:4047: use of closed network connection Aug 9 11:37:54.124866321 (18h11m48.555946492s) [seg_pos={0 0}, upload=18998]: Done waiting for segment. successful: 6 optimal: 13 errs: <nil> Aug 9 11:37:54.353990811 (18h11m48.785070942s) [upload=18998]: Done uploading segments... Aug 9 11:37:54.354028673 (18h11m48.785108804s) [upload=18998]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 6) Duration: 5.922672345s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 2 out of 60 is failed (threshold=3) [.............................................X............X.] duration: 5.557129232s FAIL (2025-08-09 11:59:39 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: too many requests ======LOGS====== Aug 9 11:58:42.586754892 (18h32m37.017835023s) [upload=19372]: Starting upload Aug 9 11:58:42.586848722 (18h32m37.017928863s) [upload=19372]: Done starting upload Aug 9 11:58:42.586948633 (18h32m37.018028754s) [upload=19372]: Uploading segments... Aug 9 11:58:42.587092008 (18h32m37.018172139s) [upload=19372]: Got next segment. Inline: false Aug 9 11:58:42.587118157 (18h32m37.018198289s) [upload=19372]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 9 11:58:42.587124269 (18h32m37.018204431s) [upload=19372]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 9 11:58:42.798975944 (18h32m37.230056075s) [upload=19372]: Next segment err: metaclient: Too Many Requests Aug 9 11:58:42.799010359 (18h32m37.23009049s) [upload=19372]: Done uploading segments... Aug 9 11:58:42.799028504 (18h32m37.230108635s) [upload=19372]: Upload finished. err: metaclient: Too Many Requests Duration: 5.557157687s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 3 out of 60 is failed (threshold=3) [..............................X..................X.........X] duration: 1.557364648s FAIL (2025-08-09 11:59:32 UTC) orbiter warning: operation: ListObjects ======ERROR====== unable to find key in bucket: sj://uptime-checks-placement-20/orbiter_us_0/1K ======LOGS====== Aug 9 11:59:31.819436107 (18h33m26.250516238s) [upload=19388]: Starting upload Aug 9 11:59:31.819492355 (18h33m26.250572516s) [upload=19388]: Done starting upload Aug 9 11:59:31.819563080 (18h33m26.250643211s) [upload=19388]: Uploading segments... Aug 9 11:59:31.819616853 (18h33m26.250696984s) [upload=19388]: Got next segment. Inline: true Aug 9 11:59:31.819621552 (18h33m26.250701683s) [upload=19388]: Waiting for error group managing segments... Aug 9 11:59:31.819660466 (18h33m26.250740597s) [upload=19388]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 9 11:59:31.819665206 (18h33m26.250745327s) [upload=19388]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 9 11:59:31.819667851 (18h33m26.250747972s) [upload=19388]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 9 11:59:32.242597809 (18h33m26.67367794s) [upload=19388]: Done uploading segments... Aug 9 11:59:32.242615142 (18h33m26.673695263s) [upload=19388]: Upload finished. err: <nil> Duration:* 1.557391729s core uplink us-select-1 us1-placement-20 metainfo-projects

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)