Orbiter History

checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [..............X.............................................] duration: 5.521710027s FAIL (2025-07-27 01:28:05 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.521728572s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [X..............X............................................] duration: 6.217939864s FAIL (2025-07-27 01:27:50 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.217964532s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [X.X.X..........X............................................] duration: 6.836720312s FAIL (2025-07-27 01:27:42 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== 7326QE72JPEXPXFHPPYHSNKIURIMRGIQ, seg_pos={0 0}, upload=58778]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR): protocol: write tcp 188.166.225.170:33562->104.36.82.11:4115: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:54:03.385284649 (61h18m13.107847679s) [seg_pos={0 0}, upload=58778]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 27 00:54:03.385367337 (61h18m13.107930377s) [upload=58778]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 27 00:54:03.385371374 (61h18m13.107934414s) [upload=58778]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 27 00:54:03.688084400 (61h18m13.41064744s) [upload=58778]: Done uploading segments... Jul 27 00:54:03.688101954 (61h18m13.410664994s) [upload=58778]: Upload finished. err: <nil> Jul 27 00:54:29.969393323 (61h18m39.691956364s) [upload=58786]: Starting upload Jul 27 00:54:29.969484158 (61h18m39.692047198s) [upload=58786]: Done starting upload Jul 27 00:54:29.969609477 (61h18m39.692172517s) [upload=58786]: Uploading segments... Jul 27 00:54:29.969791546 (61h18m39.692354576s) [upload=58786]: Got next segment. Inline: false Jul 27 00:54:29.969897859 (61h18m39.6924609s) [upload=58786]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 00:54:29.969903801 (61h18m39.692466841s) [upload=58786]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 00:54:30.202493651 (61h18m39.925056691s) [seg_pos={0 0}, upload=58786]: Begin upload segment... Jul 27 00:54:30.202544298 (61h18m39.925107338s) [seg_pos={0 0}, upload=58786]: Done begin upload segment. Jul 27 00:54:30.202565960 (61h18m39.92512899s) [noise=true , address=104.36.82.15:4259 , piece_id=2CYXGFMVV7Z4PMRHEJ23VWSK53BM3C4DRQPNSPOI55MY7FLS2OSQ, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.202629651 (61h18m39.925192691s) [noise=true , address=45.91.116.5:4299 , piece_id=5K6GYVCVQ46VZRUMNZ3PEITFO66VABEGLAV7NZCV5M36QVUVMFCA, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.202926890 (61h18m39.92548992s) [noise=true , address=45.91.116.7:4010 , piece_id=EKECSPS64QHNIO3EIRAVAT35KA5TCPBMDBLT5TSB3LZWKP6KZZHA, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.203102106 (61h18m39.925665146s) [noise=true , address=77.90.31.16:4735 , piece_id=N32CSF74OC4RGUPMJST5IV5P3FI6RBEUHKXCPXVNUFNHHSN5ZH4A, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.203204351 (61h18m39.925767391s) [noise=true , address=104.36.82.5:4838 , piece_id=JNEB4FUOK534MWNRW4DB5VJPPCUIBDA6FIX2YXWSRXNFQ6GDACHA, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.203445583 (61h18m39.926008623s) [noise=true , address=77.90.31.19:4843 , piece_id=KHZD4RVLBISBTIER5RAE2UCFCEFYYWCPMMXLSKIRUTIQTMLODQXQ, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.203491891 (61h18m39.926054921s) [noise=true , address=77.90.31.10:4520 , piece_id=4GRJDJRCNJPKJ5VV23VRZRNBLAOCJEHFVW534O3M3TQ4SRSP2FEQ, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.203718445 (61h18m39.926281475s) [noise=true , address=45.91.116.6:4974 , piece_id=LR3SRAZFX3NSX7KTZOR643UHIIJSXI4XTBCXYA6W2SFKSEBN7Q2Q, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.204010243 (61h18m39.926573283s) [noise=true , address=45.91.116.12:4227 , piece_id=J2OI4W4CT2FLAUB7WOGOAMTRWRS4PB5AQXLF4XFZNNO2J6LL46SA, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.204257396 (61h18m39.926820436s) [noise=true , address=77.90.31.12:4592 , piece_id=EA7S4T5VCRCJBRETDY6FJ43SOP7NWEVMR2F6L64J3TB6TNOOCKAA, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.204507925 (61h18m39.927070976s) [noise=true , address=45.91.116.4:4939 , piece_id=ZR6IIF5N5GSW6XIMVVRFVAE56M4HNLL5J7SSJE5M5UUFXKZFJSFQ, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.205281365 (61h18m39.927844435s) [noise=true , address=45.91.116.6:4973 , piece_id=VKQD7SENX3EDE34BXHGL4Z7MTET2OWSFLTOZ4PPYTDD2QVUHZEJQ, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.205688314 (61h18m39.928251354s) [noise=true , address=77.90.31.14:4662 , piece_id=VBVLHQKNOIU25REDYLILDDF3XRZFFU5W5IY4I7YMUE2WTZXLBHEQ, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.205981905 (61h18m39.928544945s) [noise=true , address=45.91.116.7:4009 , piece_id=4V2UC4QPMJKHBAI7BVHTVPZQFY667XVV3DCXAN3GKL444IHXEVEA, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.206218718 (61h18m39.928781759s) [noise=true , address=104.36.82.9:4005 , piece_id=3ARJN3YOZQ7Y3YYCSDQOHV5PEWB7WWJSS3WCHLJ52KWKD32CUKDQ, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.207082942 (61h18m39.929645982s) [noise=true , address=104.36.82.7:4910 , piece_id=ZJYKL4SR6BZP6OYD7QQOZADAR7GDVKVXV4GERU6FQE52RXZGCGOA, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:31.892857589 (61h18m41.615420639s) [upload=58786]: Next returned nil segment Jul 27 00:54:31.892877968 (61h18m41.615441008s) [upload=58786]: Waiting for error group managing segments... Jul 27 00:54:32.223482398 (61h18m41.946045438s) [noise=true , address=45.91.116.6:4974 , piece_id=LR3SRAZFX3NSX7KTZOR643UHIIJSXI4XTBCXYA6W2SFKSEBN7Q2Q, seg_pos={0 0}, upload=58786]: Done uploading piece. err: <nil> Jul 27 00:54:32.237060637 (61h18m41.959623677s) [noise=true , address=45.91.116.4:4939 , piece_id=ZR6IIF5N5GSW6XIMVVRFVAE56M4HNLL5J7SSJE5M5UUFXKZFJSFQ, seg_pos={0 0}, upload=58786]: Done uploading piece. err: <nil> Jul 27 00:54:32.299221539 (61h18m42.021784619s) [noise=true , address=45.91.116.7:4009 , piece_id=4V2UC4QPMJKHBAI7BVHTVPZQFY667XVV3DCXAN3GKL444IHXEVEA, seg_pos={0 0}, upload=58786]: Done uploading piece. err: <nil> Jul 27 00:54:32.325716521 (61h18m42.048279561s) [noise=true , address=45.91.116.6:4973 , piece_id=VKQD7SENX3EDE34BXHGL4Z7MTET2OWSFLTOZ4PPYTDD2QVUHZEJQ, seg_pos={0 0}, upload=58786]: Done uploading piece. err: <nil> Jul 27 00:54:32.325884473 (61h18m42.048447503s) [noise=true , address=45.91.116.12:4227 , piece_id=J2OI4W4CT2FLAUB7WOGOAMTRWRS4PB5AQXLF4XFZNNO2J6LL46SA, seg_pos={0 0}, upload=58786]: Done uploading piece. err: <nil> Jul 27 00:54:32.372983466 (61h18m42.095546506s) [noise=true , address=45.91.116.5:4299 , piece_id=5K6GYVCVQ46VZRUMNZ3PEITFO66VABEGLAV7NZCV5M36QVUVMFCA, seg_pos={0 0}, upload=58786]: Done uploading piece. err: <nil> Jul 27 00:54:32.379844103 (61h18m42.102407133s) [noise=true , address=45.91.116.7:4010 , piece_id=EKECSPS64QHNIO3EIRAVAT35KA5TCPBMDBLT5TSB3LZWKP6KZZHA, seg_pos={0 0}, upload=58786]: Done uploading piece. err: <nil> Jul 27 00:54:32.384666850 (61h18m42.10722989s) [noise=true , address=77.90.31.14:4662 , piece_id=VBVLHQKNOIU25REDYLILDDF3XRZFFU5W5IY4I7YMUE2WTZXLBHEQ, seg_pos={0 0}, upload=58786]: Done uploading piece. err: <nil> Jul 27 00:54:32.472371113 (61h18m42.194934143s) [noise=true , address=77.90.31.16:4735 , piece_id=N32CSF74OC4RGUPMJST5IV5P3FI6RBEUHKXCPXVNUFNHHSN5ZH4A, seg_pos={0 0}, upload=58786]: Done uploading piece. err: <nil> Jul 27 00:54:33.667049984 (61h18m43.389613024s) [noise=true , address=77.90.31.19:4843 , piece_id=KHZD4RVLBISBTIER5RAE2UCFCEFYYWCPMMXLSKIRUTIQTMLODQXQ, seg_pos={0 0}, upload=58786]: Done uploading piece. err: <nil> Jul 27 00:54:34.213712174 (61h18m43.936275214s) [noise=true , address=77.90.31.12:4592 , piece_id=EA7S4T5VCRCJBRETDY6FJ43SOP7NWEVMR2F6L64J3TB6TNOOCKAA, seg_pos={0 0}, upload=58786]: Done uploading piece. err: <nil> Jul 27 00:54:42.252927986 (61h18m51.975491026s) [noise=true , address=77.90.31.10:4520 , piece_id=4GRJDJRCNJPKJ5VV23VRZRNBLAOCJEHFVW534O3M3TQ4SRSP2FEQ, seg_pos={0 0}, upload=58786]: Done uploading piece. err: <nil> Jul 27 00:54:59.328639826 (61h19m9.051202866s) [noise=true , address=104.36.82.5:4838 , piece_id=JNEB4FUOK534MWNRW4DB5VJPPCUIBDA6FIX2YXWSRXNFQ6GDACHA, seg_pos={0 0}, upload=58786]: Done uploading piece. err: ecclient: upload failed (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL, address:104.36.82.5:4838): protocol: rpc: dial tcp 104.36.82.5:4838: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:54:59.328713146 (61h19m9.051276186s) [noise=true , address=104.36.82.7:4910 , piece_id=ZJYKL4SR6BZP6OYD7QQOZADAR7GDVKVXV4GERU6FQE52RXZGCGOA, seg_pos={0 0}, upload=58786]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:104.36.82.7:4910): protocol: rpc: dial tcp 104.36.82.7:4910: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:54:59.328897458 (61h19m9.051460488s) [noise=true , address=104.36.82.15:4259 , piece_id=2CYXGFMVV7Z4PMRHEJ23VWSK53BM3C4DRQPNSPOI55MY7FLS2OSQ, seg_pos={0 0}, upload=58786]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:104.36.82.15:4259): protocol: write tcp 188.166.225.170:42398->104.36.82.15:4259: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:54:59.329496955 (61h19m9.052059995s) [noise=true , address=104.36.82.9:4005 , piece_id=3ARJN3YOZQ7Y3YYCSDQOHV5PEWB7WWJSS3WCHLJ52KWKD32CUKDQ, seg_pos={0 0}, upload=58786]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:104.36.82.9:4005): protocol: rpc: dial tcp 104.36.82.9:4005: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:54:59.329528055 (61h19m9.052091095s) [seg_pos={0 0}, upload=58786]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 27 00:54:59.557689493 (61h19m9.280252533s) [upload=58786]: Done uploading segments... Jul 27 00:54:59.557710713 (61h19m9.280273753s) [upload=58786]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 6.836756791s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 4 out of 60 is failed (threshold=3) [............X....X..........X...................X...........] duration: 2.152289171s FAIL (2025-07-27 01:27:40 UTC) orbiter warning: digest mismatch: after 792f8e52f0f261131431c48786048ce73833f71a202ca313ecdd5b39aa603e1c != before 4c0418a6860f4eb95c9171181a1c1a28de1d7886deefc8fd7b2ed1a91cded834 Duration: 2.152318987s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 5 out of 60 is failed (threshold=3) [..X....X....XXX.............................................] duration: 5.815410491s FAIL (2025-07-27 01:27:32 UTC) orbiter warning: uplink: context deadline exceeded Duration: 5.81544205s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.X.X.X..........X...........................................] duration: 5.725417648s FAIL (2025-07-27 01:27:22 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== 7326QE72JPEXPXFHPPYHSNKIURIMRGIQ, seg_pos={0 0}, upload=58778]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR): protocol: write tcp 188.166.225.170:33562->104.36.82.11:4115: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:54:03.385284649 (61h18m13.107847679s) [seg_pos={0 0}, upload=58778]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 27 00:54:03.385367337 (61h18m13.107930377s) [upload=58778]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 27 00:54:03.385371374 (61h18m13.107934414s) [upload=58778]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 27 00:54:03.688084400 (61h18m13.41064744s) [upload=58778]: Done uploading segments... Jul 27 00:54:03.688101954 (61h18m13.410664994s) [upload=58778]: Upload finished. err: <nil> Jul 27 00:54:29.969393323 (61h18m39.691956364s) [upload=58786]: Starting upload Jul 27 00:54:29.969484158 (61h18m39.692047198s) [upload=58786]: Done starting upload Jul 27 00:54:29.969609477 (61h18m39.692172517s) [upload=58786]: Uploading segments... Jul 27 00:54:29.969791546 (61h18m39.692354576s) [upload=58786]: Got next segment. Inline: false Jul 27 00:54:29.969897859 (61h18m39.6924609s) [upload=58786]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 00:54:29.969903801 (61h18m39.692466841s) [upload=58786]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 00:54:30.202493651 (61h18m39.925056691s) [seg_pos={0 0}, upload=58786]: Begin upload segment... Jul 27 00:54:30.202544298 (61h18m39.925107338s) [seg_pos={0 0}, upload=58786]: Done begin upload segment. Jul 27 00:54:30.202565960 (61h18m39.92512899s) [noise=true , address=104.36.82.15:4259 , piece_id=2CYXGFMVV7Z4PMRHEJ23VWSK53BM3C4DRQPNSPOI55MY7FLS2OSQ, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.202629651 (61h18m39.925192691s) [noise=true , address=45.91.116.5:4299 , piece_id=5K6GYVCVQ46VZRUMNZ3PEITFO66VABEGLAV7NZCV5M36QVUVMFCA, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.202926890 (61h18m39.92548992s) [noise=true , address=45.91.116.7:4010 , piece_id=EKECSPS64QHNIO3EIRAVAT35KA5TCPBMDBLT5TSB3LZWKP6KZZHA, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.203102106 (61h18m39.925665146s) [noise=true , address=77.90.31.16:4735 , piece_id=N32CSF74OC4RGUPMJST5IV5P3FI6RBEUHKXCPXVNUFNHHSN5ZH4A, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.203204351 (61h18m39.925767391s) [noise=true , address=104.36.82.5:4838 , piece_id=JNEB4FUOK534MWNRW4DB5VJPPCUIBDA6FIX2YXWSRXNFQ6GDACHA, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.203445583 (61h18m39.926008623s) [noise=true , address=77.90.31.19:4843 , piece_id=KHZD4RVLBISBTIER5RAE2UCFCEFYYWCPMMXLSKIRUTIQTMLODQXQ, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.203491891 (61h18m39.926054921s) [noise=true , address=77.90.31.10:4520 , piece_id=4GRJDJRCNJPKJ5VV23VRZRNBLAOCJEHFVW534O3M3TQ4SRSP2FEQ, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.203718445 (61h18m39.926281475s) [noise=true , address=45.91.116.6:4974 , piece_id=LR3SRAZFX3NSX7KTZOR643UHIIJSXI4XTBCXYA6W2SFKSEBN7Q2Q, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.204010243 (61h18m39.926573283s) [noise=true , address=45.91.116.12:4227 , piece_id=J2OI4W4CT2FLAUB7WOGOAMTRWRS4PB5AQXLF4XFZNNO2J6LL46SA, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.204257396 (61h18m39.926820436s) [noise=true , address=77.90.31.12:4592 , piece_id=EA7S4T5VCRCJBRETDY6FJ43SOP7NWEVMR2F6L64J3TB6TNOOCKAA, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.204507925 (61h18m39.927070976s) [noise=true , address=45.91.116.4:4939 , piece_id=ZR6IIF5N5GSW6XIMVVRFVAE56M4HNLL5J7SSJE5M5UUFXKZFJSFQ, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.205281365 (61h18m39.927844435s) [noise=true , address=45.91.116.6:4973 , piece_id=VKQD7SENX3EDE34BXHGL4Z7MTET2OWSFLTOZ4PPYTDD2QVUHZEJQ, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.205688314 (61h18m39.928251354s) [noise=true , address=77.90.31.14:4662 , piece_id=VBVLHQKNOIU25REDYLILDDF3XRZFFU5W5IY4I7YMUE2WTZXLBHEQ, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.205981905 (61h18m39.928544945s) [noise=true , address=45.91.116.7:4009 , piece_id=4V2UC4QPMJKHBAI7BVHTVPZQFY667XVV3DCXAN3GKL444IHXEVEA, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.206218718 (61h18m39.928781759s) [noise=true , address=104.36.82.9:4005 , piece_id=3ARJN3YOZQ7Y3YYCSDQOHV5PEWB7WWJSS3WCHLJ52KWKD32CUKDQ, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.207082942 (61h18m39.929645982s) [noise=true , address=104.36.82.7:4910 , piece_id=ZJYKL4SR6BZP6OYD7QQOZADAR7GDVKVXV4GERU6FQE52RXZGCGOA, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:31.892857589 (61h18m41.615420639s) [upload=58786]: Next returned nil segment Jul 27 00:54:31.892877968 (61h18m41.615441008s) [upload=58786]: Waiting for error group managing segments... Jul 27 00:54:32.223482398 (61h18m41.946045438s) [noise=true , address=45.91.116.6:4974 , piece_id=LR3SRAZFX3NSX7KTZOR643UHIIJSXI4XTBCXYA6W2SFKSEBN7Q2Q, seg_pos={0 0}, upload=58786]: Done uploading piece. err: <nil> Jul 27 00:54:32.237060637 (61h18m41.959623677s) [noise=true , address=45.91.116.4:4939 , piece_id=ZR6IIF5N5GSW6XIMVVRFVAE56M4HNLL5J7SSJE5M5UUFXKZFJSFQ, seg_pos={0 0}, upload=58786]: Done uploading piece. err: <nil> Jul 27 00:54:32.299221539 (61h18m42.021784619s) [noise=true , address=45.91.116.7:4009 , piece_id=4V2UC4QPMJKHBAI7BVHTVPZQFY667XVV3DCXAN3GKL444IHXEVEA, seg_pos={0 0}, upload=58786]: Done uploading piece. err: <nil> Jul 27 00:54:32.325716521 (61h18m42.048279561s) [noise=true , address=45.91.116.6:4973 , piece_id=VKQD7SENX3EDE34BXHGL4Z7MTET2OWSFLTOZ4PPYTDD2QVUHZEJQ, seg_pos={0 0}, upload=58786]: Done uploading piece. err: <nil> Jul 27 00:54:32.325884473 (61h18m42.048447503s) [noise=true , address=45.91.116.12:4227 , piece_id=J2OI4W4CT2FLAUB7WOGOAMTRWRS4PB5AQXLF4XFZNNO2J6LL46SA, seg_pos={0 0}, upload=58786]: Done uploading piece. err: <nil> Jul 27 00:54:32.372983466 (61h18m42.095546506s) [noise=true , address=45.91.116.5:4299 , piece_id=5K6GYVCVQ46VZRUMNZ3PEITFO66VABEGLAV7NZCV5M36QVUVMFCA, seg_pos={0 0}, upload=58786]: Done uploading piece. err: <nil> Jul 27 00:54:32.379844103 (61h18m42.102407133s) [noise=true , address=45.91.116.7:4010 , piece_id=EKECSPS64QHNIO3EIRAVAT35KA5TCPBMDBLT5TSB3LZWKP6KZZHA, seg_pos={0 0}, upload=58786]: Done uploading piece. err: <nil> Jul 27 00:54:32.384666850 (61h18m42.10722989s) [noise=true , address=77.90.31.14:4662 , piece_id=VBVLHQKNOIU25REDYLILDDF3XRZFFU5W5IY4I7YMUE2WTZXLBHEQ, seg_pos={0 0}, upload=58786]: Done uploading piece. err: <nil> Jul 27 00:54:32.472371113 (61h18m42.194934143s) [noise=true , address=77.90.31.16:4735 , piece_id=N32CSF74OC4RGUPMJST5IV5P3FI6RBEUHKXCPXVNUFNHHSN5ZH4A, seg_pos={0 0}, upload=58786]: Done uploading piece. err: <nil> Jul 27 00:54:33.667049984 (61h18m43.389613024s) [noise=true , address=77.90.31.19:4843 , piece_id=KHZD4RVLBISBTIER5RAE2UCFCEFYYWCPMMXLSKIRUTIQTMLODQXQ, seg_pos={0 0}, upload=58786]: Done uploading piece. err: <nil> Jul 27 00:54:34.213712174 (61h18m43.936275214s) [noise=true , address=77.90.31.12:4592 , piece_id=EA7S4T5VCRCJBRETDY6FJ43SOP7NWEVMR2F6L64J3TB6TNOOCKAA, seg_pos={0 0}, upload=58786]: Done uploading piece. err: <nil> Jul 27 00:54:42.252927986 (61h18m51.975491026s) [noise=true , address=77.90.31.10:4520 , piece_id=4GRJDJRCNJPKJ5VV23VRZRNBLAOCJEHFVW534O3M3TQ4SRSP2FEQ, seg_pos={0 0}, upload=58786]: Done uploading piece. err: <nil> Jul 27 00:54:59.328639826 (61h19m9.051202866s) [noise=true , address=104.36.82.5:4838 , piece_id=JNEB4FUOK534MWNRW4DB5VJPPCUIBDA6FIX2YXWSRXNFQ6GDACHA, seg_pos={0 0}, upload=58786]: Done uploading piece. err: ecclient: upload failed (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL, address:104.36.82.5:4838): protocol: rpc: dial tcp 104.36.82.5:4838: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:54:59.328713146 (61h19m9.051276186s) [noise=true , address=104.36.82.7:4910 , piece_id=ZJYKL4SR6BZP6OYD7QQOZADAR7GDVKVXV4GERU6FQE52RXZGCGOA, seg_pos={0 0}, upload=58786]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:104.36.82.7:4910): protocol: rpc: dial tcp 104.36.82.7:4910: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:54:59.328897458 (61h19m9.051460488s) [noise=true , address=104.36.82.15:4259 , piece_id=2CYXGFMVV7Z4PMRHEJ23VWSK53BM3C4DRQPNSPOI55MY7FLS2OSQ, seg_pos={0 0}, upload=58786]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:104.36.82.15:4259): protocol: write tcp 188.166.225.170:42398->104.36.82.15:4259: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:54:59.329496955 (61h19m9.052059995s) [noise=true , address=104.36.82.9:4005 , piece_id=3ARJN3YOZQ7Y3YYCSDQOHV5PEWB7WWJSS3WCHLJ52KWKD32CUKDQ, seg_pos={0 0}, upload=58786]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:104.36.82.9:4005): protocol: rpc: dial tcp 104.36.82.9:4005: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:54:59.329528055 (61h19m9.052091095s) [seg_pos={0 0}, upload=58786]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 27 00:54:59.557689493 (61h19m9.280252533s) [upload=58786]: Done uploading segments... Jul 27 00:54:59.557710713 (61h19m9.280273753s) [upload=58786]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.725445021s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 8 out of 60 is failed (threshold=3) [XX.......XXXXXX.............................................] duration: 5.856043904s FAIL (2025-07-27 01:27:18 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== false Jul 27 00:54:57.821101936 (61h19m7.543664977s) [upload=58793]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 00:54:57.821108168 (61h19m7.543671208s) [upload=58793]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 00:54:58.268999381 (61h19m7.991562411s) [seg_pos={0 0}, upload=58793]: Begin upload segment... Jul 27 00:54:58.269047784 (61h19m7.991610824s) [seg_pos={0 0}, upload=58793]: Done begin upload segment. Jul 27 00:54:58.269068714 (61h19m7.991631754s) [noise=true , address=79.127.201.215:4344 , piece_id=BFA6WIJYTUHSVLKQVLXWCBKNYJJB6CXUUVTZMZEESHH4QG3OV4NQ, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.269108290 (61h19m7.99167133s) [noise=true , address=79.127.205.235:4682 , piece_id=4BXTF4IATE3AD46376DXSOM25E3UL75T4DYKSI7YKKAY4XDUGXZQ, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.269416039 (61h19m7.991979079s) [noise=true , address=104.36.82.53:4153 , piece_id=S2YS2I44RPQEKILNXGKZ6JKUWZIU47ZJTQTDDH6IKCKCEL7IM2NA, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.269588800 (61h19m7.99215184s) [noise=true , address=64.49.13.137:4502 , piece_id=COTTEJEOI7INJS3ZDBHS2CLIGMREMHZGHN7MB22TJR2Y4LZMVRRA, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.269666007 (61h19m7.992229048s) [noise=true , address=77.90.31.11:4573 , piece_id=352UN4WWJYSEOQ4VR4CDGD5XWCDY6TXNXZJW6COACYDDPNTGZSOA, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.269938439 (61h19m7.992501479s) [noise=true , address=104.36.82.46:4500 , piece_id=SQPQJFK3QRENUZNGARKNCCVDHDY2ZXIZNVOFZV76AC63YFACQTBQ, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.270186764 (61h19m7.992749794s) [noise=true , address=79.127.219.38:4001 , piece_id=ZKBCPE6IA6NQJDRGDGFQT2NNGIXAH64ZZ27OE2WPUR54XOQQ2Q3Q, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.270456489 (61h19m7.993019529s) [noise=true , address=104.36.82.47:4545 , piece_id=F4LGWDTZ3CMZIZBDGZZ4TGZ6GKFYE3GNW2B6SZEXBBMYYFS6Y3CQ, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.270693122 (61h19m7.993256162s) [noise=true , address=104.36.82.15:4289 , piece_id=MFZHE2Q766VXNQ43QOIPXO33RASSYDVTJ2QLTVR4ACCHMGJLVV2A, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.270924895 (61h19m7.993487935s) [noise=true , address=77.90.31.10:4525 , piece_id=B7APUOSESFZRV2QUHZ6KZLR7VRJTPHNZ3FFWMEPYV6O7I7EQFVDA, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.270973609 (61h19m7.993536649s) [noise=true , address=104.36.82.52:4091 , piece_id=6Z4ZKBKTEMXCSMNIC6277XZMSJPBKTFW5RJ2PC726VVJTALRJ6NA, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.271180806 (61h19m7.993743846s) [noise=true , address=79.127.205.245:4242 , piece_id=MH52WQ77OW4ZDQFVVFICHDWERB3WGGZXONOOA66J4P32ILZAREQQ, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.271367583 (61h19m7.993930613s) [noise=true , address=64.49.13.140:4514 , piece_id=RXOR5I3R3HF6WUJPKES4OQFTPMETGOEFPZB2KTUS4WVSQUIQQTDA, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.271609946 (61h19m7.994172986s) [noise=true , address=156.146.43.225:4448 , piece_id=TAPXLBQNT5ERYAWMWFBHQF4LYXAZJS4XXIAHXT57S3ORLKEX4MQQ, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.271884882 (61h19m7.994447922s) [noise=true , address=104.36.82.49:4005 , piece_id=J6YISXBRJWWSOJLDNE4U72EQFENIS7SPYZM4SCEVJG2SUGDHRNDA, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.271925130 (61h19m7.99448817s) [noise=true , address=79.127.219.47:4072 , piece_id=JBTVVQE3YIPDNKFAWIPDB5UBQZXP7KJJW4OBCXTAZDUQWLODTZWQ, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.272230464 (61h19m7.994793504s) [noise=true , address=77.90.31.16:4751 , piece_id=XC46QZVCMKTVTLQYDPGM4ZRRD4KDIFCJTQMHFDMY633FQG74FWWQ, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.272352857 (61h19m7.994915898s) [noise=true , address=79.127.220.101:4041 , piece_id=DBBBRJX5NPIPW5P72KSEANTJRZEM5MMUJQQTLXDSCYVUQY6IIIXA, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:59.403085511 (61h19m9.125648551s) [upload=58793]: Next returned nil segment Jul 27 00:54:59.403106221 (61h19m9.125669281s) [upload=58793]: Waiting for error group managing segments... Jul 27 00:54:59.699317478 (61h19m9.421880518s) [noise=true , address=64.49.13.137:4502 , piece_id=COTTEJEOI7INJS3ZDBHS2CLIGMREMHZGHN7MB22TJR2Y4LZMVRRA, seg_pos={0 0}, upload=58793]: Done uploading piece. err: <nil> Jul 27 00:54:59.712155049 (61h19m9.434718079s) [noise=true , address=79.127.205.235:4682 , piece_id=4BXTF4IATE3AD46376DXSOM25E3UL75T4DYKSI7YKKAY4XDUGXZQ, seg_pos={0 0}, upload=58793]: Done uploading piece. err: <nil> Jul 27 00:54:59.718874015 (61h19m9.441437055s) [noise=true , address=79.127.205.245:4242 , piece_id=MH52WQ77OW4ZDQFVVFICHDWERB3WGGZXONOOA66J4P32ILZAREQQ, seg_pos={0 0}, upload=58793]: Done uploading piece. err: <nil> Jul 27 00:55:00.219903397 (61h19m9.942466437s) [noise=true , address=77.90.31.16:4751 , piece_id=XC46QZVCMKTVTLQYDPGM4ZRRD4KDIFCJTQMHFDMY633FQG74FWWQ, seg_pos={0 0}, upload=58793]: Done uploading piece. err: <nil> Jul 27 00:55:00.393745135 (61h19m10.116308165s) [noise=true , address=77.90.31.11:4573 , piece_id=352UN4WWJYSEOQ4VR4CDGD5XWCDY6TXNXZJW6COACYDDPNTGZSOA, seg_pos={0 0}, upload=58793]: Done uploading piece. err: <nil> Jul 27 00:55:01.589079921 (61h19m11.311642972s) [noise=true , address=79.127.219.47:4072 , piece_id=JBTVVQE3YIPDNKFAWIPDB5UBQZXP7KJJW4OBCXTAZDUQWLODTZWQ, seg_pos={0 0}, upload=58793]: Done uploading piece. err: <nil> Jul 27 00:55:01.684825540 (61h19m11.40738858s) [noise=true , address=79.127.220.101:4041 , piece_id=DBBBRJX5NPIPW5P72KSEANTJRZEM5MMUJQQTLXDSCYVUQY6IIIXA, seg_pos={0 0}, upload=58793]: Done uploading piece. err: <nil> Jul 27 00:55:02.296389006 (61h19m12.018952046s) [noise=true , address=156.146.43.225:4448 , piece_id=TAPXLBQNT5ERYAWMWFBHQF4LYXAZJS4XXIAHXT57S3ORLKEX4MQQ, seg_pos={0 0}, upload=58793]: Done uploading piece. err: <nil> Jul 27 00:55:03.015602295 (61h19m12.738165335s) [noise=true , address=79.127.219.38:4001 , piece_id=ZKBCPE6IA6NQJDRGDGFQT2NNGIXAH64ZZ27OE2WPUR54XOQQ2Q3Q, seg_pos={0 0}, upload=58793]: Done uploading piece. err: <nil> Jul 27 00:55:03.588380603 (61h19m13.310943643s) [noise=true , address=77.90.31.10:4525 , piece_id=B7APUOSESFZRV2QUHZ6KZLR7VRJTPHNZ3FFWMEPYV6O7I7EQFVDA, seg_pos={0 0}, upload=58793]: Done uploading piece. err: <nil> Jul 27 00:55:07.419707501 (61h19m17.142270581s) [noise=true , address=79.127.201.215:4344 , piece_id=BFA6WIJYTUHSVLKQVLXWCBKNYJJB6CXUUVTZMZEESHH4QG3OV4NQ, seg_pos={0 0}, upload=58793]: Done uploading piece. err: <nil> Jul 27 00:55:10.064731586 (61h19m19.787294626s) [noise=true , address=64.49.13.140:4514 , piece_id=RXOR5I3R3HF6WUJPKES4OQFTPMETGOEFPZB2KTUS4WVSQUIQQTDA, seg_pos={0 0}, upload=58793]: Done uploading piece. err: <nil> Jul 27 00:55:57.170253087 (61h20m6.892816127s) [noise=true , address=104.36.82.53:4153 , piece_id=S2YS2I44RPQEKILNXGKZ6JKUWZIU47ZJTQTDDH6IKCKCEL7IM2NA, seg_pos={0 0}, upload=58793]: Done uploading piece. err: ecclient: upload failed (node:1fPp4DvLe18dZWPmMPFiC8o9jL4uiEFNmPAHHmFEH74ZXNndw3, address:104.36.82.53:4153): protocol: write tcp 188.166.225.170:41520->104.36.82.53:4153: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:55:57.170329522 (61h20m6.892892562s) [noise=true , address=104.36.82.47:4545 , piece_id=F4LGWDTZ3CMZIZBDGZZ4TGZ6GKFYE3GNW2B6SZEXBBMYYFS6Y3CQ, seg_pos={0 0}, upload=58793]: Done uploading piece. err: ecclient: upload failed (node:12euvmhGegnPYufoKKQ1on9FTv2AhuicSVYuWyXtDiY9BLdDMYE, address:104.36.82.47:4545): protocol: rpc: dial tcp 104.36.82.47:4545: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:55:57.170370181 (61h20m6.892933221s) [noise=true , address=104.36.82.52:4091 , piece_id=6Z4ZKBKTEMXCSMNIC6277XZMSJPBKTFW5RJ2PC726VVJTALRJ6NA, seg_pos={0 0}, upload=58793]: Done uploading piece. err: ecclient: upload failed (node:17wSQzrFc2hUimSsNwDxCgDDi4TpHkbCEH38fGKf9B8WqWzxpW, address:104.36.82.52:4091): protocol: rpc: dial tcp 104.36.82.52:4091: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:55:57.170401270 (61h20m6.89296431s) [noise=true , address=104.36.82.46:4500 , piece_id=SQPQJFK3QRENUZNGARKNCCVDHDY2ZXIZNVOFZV76AC63YFACQTBQ, seg_pos={0 0}, upload=58793]: Done uploading piece. err: ecclient: upload failed (node:12B7ox4ehqxJ2mwqeXj8C9N9GuSU5etNC23iT1pJnTgMLGBAGQK, address:104.36.82.46:4500): protocol: rpc: dial tcp 104.36.82.46:4500: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:55:57.170514156 (61h20m6.893077196s) [noise=true , address=104.36.82.49:4005 , piece_id=J6YISXBRJWWSOJLDNE4U72EQFENIS7SPYZM4SCEVJG2SUGDHRNDA, seg_pos={0 0}, upload=58793]: Done uploading piece. err: ecclient: upload failed (node:1AivThSVLAp9zneNK79sic4AqnTQ62yvNTFF36imigarMGqDSW, address:104.36.82.49:4005): protocol: rpc: dial tcp 104.36.82.49:4005: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:55:57.170524104 (61h20m6.893087155s) [noise=true , address=104.36.82.15:4289 , piece_id=MFZHE2Q766VXNQ43QOIPXO33RASSYDVTJ2QLTVR4ACCHMGJLVV2A, seg_pos={0 0}, upload=58793]: Done uploading piece. err: ecclient: upload failed (node:126WPF6MtX7YjuKf73rGJrifdXyd78KY4FdL757qDipGNHBW6uY, address:104.36.82.15:4289): protocol: rpc: dial tcp 104.36.82.15:4289: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:55:57.170556958 (61h20m6.893119998s) [seg_pos={0 0}, upload=58793]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Jul 27 00:55:57.400256958 (61h20m7.122819998s) [upload=58793]: Done uploading segments... Jul 27 00:55:57.400281876 (61h20m7.122844916s) [upload=58793]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 5.856084732s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [.X..............X...........................................] duration: 5.655307588s FAIL (2025-07-27 01:27:05 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.65533024s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 4 out of 60 is failed (threshold=3) [.............X....X..........X...................X..........] duration: 1.913998844s FAIL (2025-07-27 01:27:03 UTC) orbiter warning: digest mismatch: after 792f8e52f0f261131431c48786048ce73833f71a202ca313ecdd5b39aa603e1c != before 4c0418a6860f4eb95c9171181a1c1a28de1d7886deefc8fd7b2ed1a91cded834 Duration: 1.914029552s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [..X.X.X..........X..........................................] duration: 5.784959188s FAIL (2025-07-27 01:26:50 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== 7326QE72JPEXPXFHPPYHSNKIURIMRGIQ, seg_pos={0 0}, upload=58778]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR): protocol: write tcp 188.166.225.170:33562->104.36.82.11:4115: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:54:03.385284649 (61h18m13.107847679s) [seg_pos={0 0}, upload=58778]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 27 00:54:03.385367337 (61h18m13.107930377s) [upload=58778]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 27 00:54:03.385371374 (61h18m13.107934414s) [upload=58778]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 27 00:54:03.688084400 (61h18m13.41064744s) [upload=58778]: Done uploading segments... Jul 27 00:54:03.688101954 (61h18m13.410664994s) [upload=58778]: Upload finished. err: <nil> Jul 27 00:54:29.969393323 (61h18m39.691956364s) [upload=58786]: Starting upload Jul 27 00:54:29.969484158 (61h18m39.692047198s) [upload=58786]: Done starting upload Jul 27 00:54:29.969609477 (61h18m39.692172517s) [upload=58786]: Uploading segments... Jul 27 00:54:29.969791546 (61h18m39.692354576s) [upload=58786]: Got next segment. Inline: false Jul 27 00:54:29.969897859 (61h18m39.6924609s) [upload=58786]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 00:54:29.969903801 (61h18m39.692466841s) [upload=58786]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 00:54:30.202493651 (61h18m39.925056691s) [seg_pos={0 0}, upload=58786]: Begin upload segment... Jul 27 00:54:30.202544298 (61h18m39.925107338s) [seg_pos={0 0}, upload=58786]: Done begin upload segment. Jul 27 00:54:30.202565960 (61h18m39.92512899s) [noise=true , address=104.36.82.15:4259 , piece_id=2CYXGFMVV7Z4PMRHEJ23VWSK53BM3C4DRQPNSPOI55MY7FLS2OSQ, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.202629651 (61h18m39.925192691s) [noise=true , address=45.91.116.5:4299 , piece_id=5K6GYVCVQ46VZRUMNZ3PEITFO66VABEGLAV7NZCV5M36QVUVMFCA, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.202926890 (61h18m39.92548992s) [noise=true , address=45.91.116.7:4010 , piece_id=EKECSPS64QHNIO3EIRAVAT35KA5TCPBMDBLT5TSB3LZWKP6KZZHA, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.203102106 (61h18m39.925665146s) [noise=true , address=77.90.31.16:4735 , piece_id=N32CSF74OC4RGUPMJST5IV5P3FI6RBEUHKXCPXVNUFNHHSN5ZH4A, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.203204351 (61h18m39.925767391s) [noise=true , address=104.36.82.5:4838 , piece_id=JNEB4FUOK534MWNRW4DB5VJPPCUIBDA6FIX2YXWSRXNFQ6GDACHA, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.203445583 (61h18m39.926008623s) [noise=true , address=77.90.31.19:4843 , piece_id=KHZD4RVLBISBTIER5RAE2UCFCEFYYWCPMMXLSKIRUTIQTMLODQXQ, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.203491891 (61h18m39.926054921s) [noise=true , address=77.90.31.10:4520 , piece_id=4GRJDJRCNJPKJ5VV23VRZRNBLAOCJEHFVW534O3M3TQ4SRSP2FEQ, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.203718445 (61h18m39.926281475s) [noise=true , address=45.91.116.6:4974 , piece_id=LR3SRAZFX3NSX7KTZOR643UHIIJSXI4XTBCXYA6W2SFKSEBN7Q2Q, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.204010243 (61h18m39.926573283s) [noise=true , address=45.91.116.12:4227 , piece_id=J2OI4W4CT2FLAUB7WOGOAMTRWRS4PB5AQXLF4XFZNNO2J6LL46SA, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.204257396 (61h18m39.926820436s) [noise=true , address=77.90.31.12:4592 , piece_id=EA7S4T5VCRCJBRETDY6FJ43SOP7NWEVMR2F6L64J3TB6TNOOCKAA, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.204507925 (61h18m39.927070976s) [noise=true , address=45.91.116.4:4939 , piece_id=ZR6IIF5N5GSW6XIMVVRFVAE56M4HNLL5J7SSJE5M5UUFXKZFJSFQ, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.205281365 (61h18m39.927844435s) [noise=true , address=45.91.116.6:4973 , piece_id=VKQD7SENX3EDE34BXHGL4Z7MTET2OWSFLTOZ4PPYTDD2QVUHZEJQ, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.205688314 (61h18m39.928251354s) [noise=true , address=77.90.31.14:4662 , piece_id=VBVLHQKNOIU25REDYLILDDF3XRZFFU5W5IY4I7YMUE2WTZXLBHEQ, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.205981905 (61h18m39.928544945s) [noise=true , address=45.91.116.7:4009 , piece_id=4V2UC4QPMJKHBAI7BVHTVPZQFY667XVV3DCXAN3GKL444IHXEVEA, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.206218718 (61h18m39.928781759s) [noise=true , address=104.36.82.9:4005 , piece_id=3ARJN3YOZQ7Y3YYCSDQOHV5PEWB7WWJSS3WCHLJ52KWKD32CUKDQ, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.207082942 (61h18m39.929645982s) [noise=true , address=104.36.82.7:4910 , piece_id=ZJYKL4SR6BZP6OYD7QQOZADAR7GDVKVXV4GERU6FQE52RXZGCGOA, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:31.892857589 (61h18m41.615420639s) [upload=58786]: Next returned nil segment Jul 27 00:54:31.892877968 (61h18m41.615441008s) [upload=58786]: Waiting for error group managing segments... Jul 27 00:54:32.223482398 (61h18m41.946045438s) [noise=true , address=45.91.116.6:4974 , piece_id=LR3SRAZFX3NSX7KTZOR643UHIIJSXI4XTBCXYA6W2SFKSEBN7Q2Q, seg_pos={0 0}, upload=58786]: Done uploading piece. err: <nil> Jul 27 00:54:32.237060637 (61h18m41.959623677s) [noise=true , address=45.91.116.4:4939 , piece_id=ZR6IIF5N5GSW6XIMVVRFVAE56M4HNLL5J7SSJE5M5UUFXKZFJSFQ, seg_pos={0 0}, upload=58786]: Done uploading piece. err: <nil> Jul 27 00:54:32.299221539 (61h18m42.021784619s) [noise=true , address=45.91.116.7:4009 , piece_id=4V2UC4QPMJKHBAI7BVHTVPZQFY667XVV3DCXAN3GKL444IHXEVEA, seg_pos={0 0}, upload=58786]: Done uploading piece. err: <nil> Jul 27 00:54:32.325716521 (61h18m42.048279561s) [noise=true , address=45.91.116.6:4973 , piece_id=VKQD7SENX3EDE34BXHGL4Z7MTET2OWSFLTOZ4PPYTDD2QVUHZEJQ, seg_pos={0 0}, upload=58786]: Done uploading piece. err: <nil> Jul 27 00:54:32.325884473 (61h18m42.048447503s) [noise=true , address=45.91.116.12:4227 , piece_id=J2OI4W4CT2FLAUB7WOGOAMTRWRS4PB5AQXLF4XFZNNO2J6LL46SA, seg_pos={0 0}, upload=58786]: Done uploading piece. err: <nil> Jul 27 00:54:32.372983466 (61h18m42.095546506s) [noise=true , address=45.91.116.5:4299 , piece_id=5K6GYVCVQ46VZRUMNZ3PEITFO66VABEGLAV7NZCV5M36QVUVMFCA, seg_pos={0 0}, upload=58786]: Done uploading piece. err: <nil> Jul 27 00:54:32.379844103 (61h18m42.102407133s) [noise=true , address=45.91.116.7:4010 , piece_id=EKECSPS64QHNIO3EIRAVAT35KA5TCPBMDBLT5TSB3LZWKP6KZZHA, seg_pos={0 0}, upload=58786]: Done uploading piece. err: <nil> Jul 27 00:54:32.384666850 (61h18m42.10722989s) [noise=true , address=77.90.31.14:4662 , piece_id=VBVLHQKNOIU25REDYLILDDF3XRZFFU5W5IY4I7YMUE2WTZXLBHEQ, seg_pos={0 0}, upload=58786]: Done uploading piece. err: <nil> Jul 27 00:54:32.472371113 (61h18m42.194934143s) [noise=true , address=77.90.31.16:4735 , piece_id=N32CSF74OC4RGUPMJST5IV5P3FI6RBEUHKXCPXVNUFNHHSN5ZH4A, seg_pos={0 0}, upload=58786]: Done uploading piece. err: <nil> Jul 27 00:54:33.667049984 (61h18m43.389613024s) [noise=true , address=77.90.31.19:4843 , piece_id=KHZD4RVLBISBTIER5RAE2UCFCEFYYWCPMMXLSKIRUTIQTMLODQXQ, seg_pos={0 0}, upload=58786]: Done uploading piece. err: <nil> Jul 27 00:54:34.213712174 (61h18m43.936275214s) [noise=true , address=77.90.31.12:4592 , piece_id=EA7S4T5VCRCJBRETDY6FJ43SOP7NWEVMR2F6L64J3TB6TNOOCKAA, seg_pos={0 0}, upload=58786]: Done uploading piece. err: <nil> Jul 27 00:54:42.252927986 (61h18m51.975491026s) [noise=true , address=77.90.31.10:4520 , piece_id=4GRJDJRCNJPKJ5VV23VRZRNBLAOCJEHFVW534O3M3TQ4SRSP2FEQ, seg_pos={0 0}, upload=58786]: Done uploading piece. err: <nil> Jul 27 00:54:59.328639826 (61h19m9.051202866s) [noise=true , address=104.36.82.5:4838 , piece_id=JNEB4FUOK534MWNRW4DB5VJPPCUIBDA6FIX2YXWSRXNFQ6GDACHA, seg_pos={0 0}, upload=58786]: Done uploading piece. err: ecclient: upload failed (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL, address:104.36.82.5:4838): protocol: rpc: dial tcp 104.36.82.5:4838: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:54:59.328713146 (61h19m9.051276186s) [noise=true , address=104.36.82.7:4910 , piece_id=ZJYKL4SR6BZP6OYD7QQOZADAR7GDVKVXV4GERU6FQE52RXZGCGOA, seg_pos={0 0}, upload=58786]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:104.36.82.7:4910): protocol: rpc: dial tcp 104.36.82.7:4910: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:54:59.328897458 (61h19m9.051460488s) [noise=true , address=104.36.82.15:4259 , piece_id=2CYXGFMVV7Z4PMRHEJ23VWSK53BM3C4DRQPNSPOI55MY7FLS2OSQ, seg_pos={0 0}, upload=58786]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:104.36.82.15:4259): protocol: write tcp 188.166.225.170:42398->104.36.82.15:4259: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:54:59.329496955 (61h19m9.052059995s) [noise=true , address=104.36.82.9:4005 , piece_id=3ARJN3YOZQ7Y3YYCSDQOHV5PEWB7WWJSS3WCHLJ52KWKD32CUKDQ, seg_pos={0 0}, upload=58786]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:104.36.82.9:4005): protocol: rpc: dial tcp 104.36.82.9:4005: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:54:59.329528055 (61h19m9.052091095s) [seg_pos={0 0}, upload=58786]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 27 00:54:59.557689493 (61h19m9.280252533s) [upload=58786]: Done uploading segments... Jul 27 00:54:59.557710713 (61h19m9.280273753s) [upload=58786]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.784989577s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 5 out of 60 is failed (threshold=3) [...X....X....XXX............................................] duration: 5.80251566s FAIL (2025-07-27 01:26:27 UTC) orbiter warning: uplink: context deadline exceeded Duration: 5.802543563s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 9 out of 60 is failed (threshold=3) [XXX.......XXXXXX............................................] duration: 5.502825966s FAIL (2025-07-27 01:26:23 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== false Jul 27 00:54:57.821101936 (61h19m7.543664977s) [upload=58793]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 00:54:57.821108168 (61h19m7.543671208s) [upload=58793]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 00:54:58.268999381 (61h19m7.991562411s) [seg_pos={0 0}, upload=58793]: Begin upload segment... Jul 27 00:54:58.269047784 (61h19m7.991610824s) [seg_pos={0 0}, upload=58793]: Done begin upload segment. Jul 27 00:54:58.269068714 (61h19m7.991631754s) [noise=true , address=79.127.201.215:4344 , piece_id=BFA6WIJYTUHSVLKQVLXWCBKNYJJB6CXUUVTZMZEESHH4QG3OV4NQ, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.269108290 (61h19m7.99167133s) [noise=true , address=79.127.205.235:4682 , piece_id=4BXTF4IATE3AD46376DXSOM25E3UL75T4DYKSI7YKKAY4XDUGXZQ, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.269416039 (61h19m7.991979079s) [noise=true , address=104.36.82.53:4153 , piece_id=S2YS2I44RPQEKILNXGKZ6JKUWZIU47ZJTQTDDH6IKCKCEL7IM2NA, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.269588800 (61h19m7.99215184s) [noise=true , address=64.49.13.137:4502 , piece_id=COTTEJEOI7INJS3ZDBHS2CLIGMREMHZGHN7MB22TJR2Y4LZMVRRA, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.269666007 (61h19m7.992229048s) [noise=true , address=77.90.31.11:4573 , piece_id=352UN4WWJYSEOQ4VR4CDGD5XWCDY6TXNXZJW6COACYDDPNTGZSOA, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.269938439 (61h19m7.992501479s) [noise=true , address=104.36.82.46:4500 , piece_id=SQPQJFK3QRENUZNGARKNCCVDHDY2ZXIZNVOFZV76AC63YFACQTBQ, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.270186764 (61h19m7.992749794s) [noise=true , address=79.127.219.38:4001 , piece_id=ZKBCPE6IA6NQJDRGDGFQT2NNGIXAH64ZZ27OE2WPUR54XOQQ2Q3Q, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.270456489 (61h19m7.993019529s) [noise=true , address=104.36.82.47:4545 , piece_id=F4LGWDTZ3CMZIZBDGZZ4TGZ6GKFYE3GNW2B6SZEXBBMYYFS6Y3CQ, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.270693122 (61h19m7.993256162s) [noise=true , address=104.36.82.15:4289 , piece_id=MFZHE2Q766VXNQ43QOIPXO33RASSYDVTJ2QLTVR4ACCHMGJLVV2A, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.270924895 (61h19m7.993487935s) [noise=true , address=77.90.31.10:4525 , piece_id=B7APUOSESFZRV2QUHZ6KZLR7VRJTPHNZ3FFWMEPYV6O7I7EQFVDA, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.270973609 (61h19m7.993536649s) [noise=true , address=104.36.82.52:4091 , piece_id=6Z4ZKBKTEMXCSMNIC6277XZMSJPBKTFW5RJ2PC726VVJTALRJ6NA, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.271180806 (61h19m7.993743846s) [noise=true , address=79.127.205.245:4242 , piece_id=MH52WQ77OW4ZDQFVVFICHDWERB3WGGZXONOOA66J4P32ILZAREQQ, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.271367583 (61h19m7.993930613s) [noise=true , address=64.49.13.140:4514 , piece_id=RXOR5I3R3HF6WUJPKES4OQFTPMETGOEFPZB2KTUS4WVSQUIQQTDA, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.271609946 (61h19m7.994172986s) [noise=true , address=156.146.43.225:4448 , piece_id=TAPXLBQNT5ERYAWMWFBHQF4LYXAZJS4XXIAHXT57S3ORLKEX4MQQ, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.271884882 (61h19m7.994447922s) [noise=true , address=104.36.82.49:4005 , piece_id=J6YISXBRJWWSOJLDNE4U72EQFENIS7SPYZM4SCEVJG2SUGDHRNDA, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.271925130 (61h19m7.99448817s) [noise=true , address=79.127.219.47:4072 , piece_id=JBTVVQE3YIPDNKFAWIPDB5UBQZXP7KJJW4OBCXTAZDUQWLODTZWQ, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.272230464 (61h19m7.994793504s) [noise=true , address=77.90.31.16:4751 , piece_id=XC46QZVCMKTVTLQYDPGM4ZRRD4KDIFCJTQMHFDMY633FQG74FWWQ, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.272352857 (61h19m7.994915898s) [noise=true , address=79.127.220.101:4041 , piece_id=DBBBRJX5NPIPW5P72KSEANTJRZEM5MMUJQQTLXDSCYVUQY6IIIXA, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:59.403085511 (61h19m9.125648551s) [upload=58793]: Next returned nil segment Jul 27 00:54:59.403106221 (61h19m9.125669281s) [upload=58793]: Waiting for error group managing segments... Jul 27 00:54:59.699317478 (61h19m9.421880518s) [noise=true , address=64.49.13.137:4502 , piece_id=COTTEJEOI7INJS3ZDBHS2CLIGMREMHZGHN7MB22TJR2Y4LZMVRRA, seg_pos={0 0}, upload=58793]: Done uploading piece. err: <nil> Jul 27 00:54:59.712155049 (61h19m9.434718079s) [noise=true , address=79.127.205.235:4682 , piece_id=4BXTF4IATE3AD46376DXSOM25E3UL75T4DYKSI7YKKAY4XDUGXZQ, seg_pos={0 0}, upload=58793]: Done uploading piece. err: <nil> Jul 27 00:54:59.718874015 (61h19m9.441437055s) [noise=true , address=79.127.205.245:4242 , piece_id=MH52WQ77OW4ZDQFVVFICHDWERB3WGGZXONOOA66J4P32ILZAREQQ, seg_pos={0 0}, upload=58793]: Done uploading piece. err: <nil> Jul 27 00:55:00.219903397 (61h19m9.942466437s) [noise=true , address=77.90.31.16:4751 , piece_id=XC46QZVCMKTVTLQYDPGM4ZRRD4KDIFCJTQMHFDMY633FQG74FWWQ, seg_pos={0 0}, upload=58793]: Done uploading piece. err: <nil> Jul 27 00:55:00.393745135 (61h19m10.116308165s) [noise=true , address=77.90.31.11:4573 , piece_id=352UN4WWJYSEOQ4VR4CDGD5XWCDY6TXNXZJW6COACYDDPNTGZSOA, seg_pos={0 0}, upload=58793]: Done uploading piece. err: <nil> Jul 27 00:55:01.589079921 (61h19m11.311642972s) [noise=true , address=79.127.219.47:4072 , piece_id=JBTVVQE3YIPDNKFAWIPDB5UBQZXP7KJJW4OBCXTAZDUQWLODTZWQ, seg_pos={0 0}, upload=58793]: Done uploading piece. err: <nil> Jul 27 00:55:01.684825540 (61h19m11.40738858s) [noise=true , address=79.127.220.101:4041 , piece_id=DBBBRJX5NPIPW5P72KSEANTJRZEM5MMUJQQTLXDSCYVUQY6IIIXA, seg_pos={0 0}, upload=58793]: Done uploading piece. err: <nil> Jul 27 00:55:02.296389006 (61h19m12.018952046s) [noise=true , address=156.146.43.225:4448 , piece_id=TAPXLBQNT5ERYAWMWFBHQF4LYXAZJS4XXIAHXT57S3ORLKEX4MQQ, seg_pos={0 0}, upload=58793]: Done uploading piece. err: <nil> Jul 27 00:55:03.015602295 (61h19m12.738165335s) [noise=true , address=79.127.219.38:4001 , piece_id=ZKBCPE6IA6NQJDRGDGFQT2NNGIXAH64ZZ27OE2WPUR54XOQQ2Q3Q, seg_pos={0 0}, upload=58793]: Done uploading piece. err: <nil> Jul 27 00:55:03.588380603 (61h19m13.310943643s) [noise=true , address=77.90.31.10:4525 , piece_id=B7APUOSESFZRV2QUHZ6KZLR7VRJTPHNZ3FFWMEPYV6O7I7EQFVDA, seg_pos={0 0}, upload=58793]: Done uploading piece. err: <nil> Jul 27 00:55:07.419707501 (61h19m17.142270581s) [noise=true , address=79.127.201.215:4344 , piece_id=BFA6WIJYTUHSVLKQVLXWCBKNYJJB6CXUUVTZMZEESHH4QG3OV4NQ, seg_pos={0 0}, upload=58793]: Done uploading piece. err: <nil> Jul 27 00:55:10.064731586 (61h19m19.787294626s) [noise=true , address=64.49.13.140:4514 , piece_id=RXOR5I3R3HF6WUJPKES4OQFTPMETGOEFPZB2KTUS4WVSQUIQQTDA, seg_pos={0 0}, upload=58793]: Done uploading piece. err: <nil> Jul 27 00:55:57.170253087 (61h20m6.892816127s) [noise=true , address=104.36.82.53:4153 , piece_id=S2YS2I44RPQEKILNXGKZ6JKUWZIU47ZJTQTDDH6IKCKCEL7IM2NA, seg_pos={0 0}, upload=58793]: Done uploading piece. err: ecclient: upload failed (node:1fPp4DvLe18dZWPmMPFiC8o9jL4uiEFNmPAHHmFEH74ZXNndw3, address:104.36.82.53:4153): protocol: write tcp 188.166.225.170:41520->104.36.82.53:4153: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:55:57.170329522 (61h20m6.892892562s) [noise=true , address=104.36.82.47:4545 , piece_id=F4LGWDTZ3CMZIZBDGZZ4TGZ6GKFYE3GNW2B6SZEXBBMYYFS6Y3CQ, seg_pos={0 0}, upload=58793]: Done uploading piece. err: ecclient: upload failed (node:12euvmhGegnPYufoKKQ1on9FTv2AhuicSVYuWyXtDiY9BLdDMYE, address:104.36.82.47:4545): protocol: rpc: dial tcp 104.36.82.47:4545: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:55:57.170370181 (61h20m6.892933221s) [noise=true , address=104.36.82.52:4091 , piece_id=6Z4ZKBKTEMXCSMNIC6277XZMSJPBKTFW5RJ2PC726VVJTALRJ6NA, seg_pos={0 0}, upload=58793]: Done uploading piece. err: ecclient: upload failed (node:17wSQzrFc2hUimSsNwDxCgDDi4TpHkbCEH38fGKf9B8WqWzxpW, address:104.36.82.52:4091): protocol: rpc: dial tcp 104.36.82.52:4091: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:55:57.170401270 (61h20m6.89296431s) [noise=true , address=104.36.82.46:4500 , piece_id=SQPQJFK3QRENUZNGARKNCCVDHDY2ZXIZNVOFZV76AC63YFACQTBQ, seg_pos={0 0}, upload=58793]: Done uploading piece. err: ecclient: upload failed (node:12B7ox4ehqxJ2mwqeXj8C9N9GuSU5etNC23iT1pJnTgMLGBAGQK, address:104.36.82.46:4500): protocol: rpc: dial tcp 104.36.82.46:4500: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:55:57.170514156 (61h20m6.893077196s) [noise=true , address=104.36.82.49:4005 , piece_id=J6YISXBRJWWSOJLDNE4U72EQFENIS7SPYZM4SCEVJG2SUGDHRNDA, seg_pos={0 0}, upload=58793]: Done uploading piece. err: ecclient: upload failed (node:1AivThSVLAp9zneNK79sic4AqnTQ62yvNTFF36imigarMGqDSW, address:104.36.82.49:4005): protocol: rpc: dial tcp 104.36.82.49:4005: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:55:57.170524104 (61h20m6.893087155s) [noise=true , address=104.36.82.15:4289 , piece_id=MFZHE2Q766VXNQ43QOIPXO33RASSYDVTJ2QLTVR4ACCHMGJLVV2A, seg_pos={0 0}, upload=58793]: Done uploading piece. err: ecclient: upload failed (node:126WPF6MtX7YjuKf73rGJrifdXyd78KY4FdL757qDipGNHBW6uY, address:104.36.82.15:4289): protocol: rpc: dial tcp 104.36.82.15:4289: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:55:57.170556958 (61h20m6.893119998s) [seg_pos={0 0}, upload=58793]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Jul 27 00:55:57.400256958 (61h20m7.122819998s) [upload=58793]: Done uploading segments... Jul 27 00:55:57.400281876 (61h20m7.122844916s) [upload=58793]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 5.502863048s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [X.X..............X..........................................] duration: 5.640064352s FAIL (2025-07-27 01:26:20 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.640084921s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [X..X.X.X..........X.........................................] duration: 5.848303174s FAIL (2025-07-27 01:26:07 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== 7326QE72JPEXPXFHPPYHSNKIURIMRGIQ, seg_pos={0 0}, upload=58778]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR): protocol: write tcp 188.166.225.170:33562->104.36.82.11:4115: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:54:03.385284649 (61h18m13.107847679s) [seg_pos={0 0}, upload=58778]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 27 00:54:03.385367337 (61h18m13.107930377s) [upload=58778]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 27 00:54:03.385371374 (61h18m13.107934414s) [upload=58778]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 27 00:54:03.688084400 (61h18m13.41064744s) [upload=58778]: Done uploading segments... Jul 27 00:54:03.688101954 (61h18m13.410664994s) [upload=58778]: Upload finished. err: <nil> Jul 27 00:54:29.969393323 (61h18m39.691956364s) [upload=58786]: Starting upload Jul 27 00:54:29.969484158 (61h18m39.692047198s) [upload=58786]: Done starting upload Jul 27 00:54:29.969609477 (61h18m39.692172517s) [upload=58786]: Uploading segments... Jul 27 00:54:29.969791546 (61h18m39.692354576s) [upload=58786]: Got next segment. Inline: false Jul 27 00:54:29.969897859 (61h18m39.6924609s) [upload=58786]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 00:54:29.969903801 (61h18m39.692466841s) [upload=58786]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 00:54:30.202493651 (61h18m39.925056691s) [seg_pos={0 0}, upload=58786]: Begin upload segment... Jul 27 00:54:30.202544298 (61h18m39.925107338s) [seg_pos={0 0}, upload=58786]: Done begin upload segment. Jul 27 00:54:30.202565960 (61h18m39.92512899s) [noise=true , address=104.36.82.15:4259 , piece_id=2CYXGFMVV7Z4PMRHEJ23VWSK53BM3C4DRQPNSPOI55MY7FLS2OSQ, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.202629651 (61h18m39.925192691s) [noise=true , address=45.91.116.5:4299 , piece_id=5K6GYVCVQ46VZRUMNZ3PEITFO66VABEGLAV7NZCV5M36QVUVMFCA, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.202926890 (61h18m39.92548992s) [noise=true , address=45.91.116.7:4010 , piece_id=EKECSPS64QHNIO3EIRAVAT35KA5TCPBMDBLT5TSB3LZWKP6KZZHA, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.203102106 (61h18m39.925665146s) [noise=true , address=77.90.31.16:4735 , piece_id=N32CSF74OC4RGUPMJST5IV5P3FI6RBEUHKXCPXVNUFNHHSN5ZH4A, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.203204351 (61h18m39.925767391s) [noise=true , address=104.36.82.5:4838 , piece_id=JNEB4FUOK534MWNRW4DB5VJPPCUIBDA6FIX2YXWSRXNFQ6GDACHA, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.203445583 (61h18m39.926008623s) [noise=true , address=77.90.31.19:4843 , piece_id=KHZD4RVLBISBTIER5RAE2UCFCEFYYWCPMMXLSKIRUTIQTMLODQXQ, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.203491891 (61h18m39.926054921s) [noise=true , address=77.90.31.10:4520 , piece_id=4GRJDJRCNJPKJ5VV23VRZRNBLAOCJEHFVW534O3M3TQ4SRSP2FEQ, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.203718445 (61h18m39.926281475s) [noise=true , address=45.91.116.6:4974 , piece_id=LR3SRAZFX3NSX7KTZOR643UHIIJSXI4XTBCXYA6W2SFKSEBN7Q2Q, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.204010243 (61h18m39.926573283s) [noise=true , address=45.91.116.12:4227 , piece_id=J2OI4W4CT2FLAUB7WOGOAMTRWRS4PB5AQXLF4XFZNNO2J6LL46SA, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.204257396 (61h18m39.926820436s) [noise=true , address=77.90.31.12:4592 , piece_id=EA7S4T5VCRCJBRETDY6FJ43SOP7NWEVMR2F6L64J3TB6TNOOCKAA, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.204507925 (61h18m39.927070976s) [noise=true , address=45.91.116.4:4939 , piece_id=ZR6IIF5N5GSW6XIMVVRFVAE56M4HNLL5J7SSJE5M5UUFXKZFJSFQ, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.205281365 (61h18m39.927844435s) [noise=true , address=45.91.116.6:4973 , piece_id=VKQD7SENX3EDE34BXHGL4Z7MTET2OWSFLTOZ4PPYTDD2QVUHZEJQ, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.205688314 (61h18m39.928251354s) [noise=true , address=77.90.31.14:4662 , piece_id=VBVLHQKNOIU25REDYLILDDF3XRZFFU5W5IY4I7YMUE2WTZXLBHEQ, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.205981905 (61h18m39.928544945s) [noise=true , address=45.91.116.7:4009 , piece_id=4V2UC4QPMJKHBAI7BVHTVPZQFY667XVV3DCXAN3GKL444IHXEVEA, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.206218718 (61h18m39.928781759s) [noise=true , address=104.36.82.9:4005 , piece_id=3ARJN3YOZQ7Y3YYCSDQOHV5PEWB7WWJSS3WCHLJ52KWKD32CUKDQ, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.207082942 (61h18m39.929645982s) [noise=true , address=104.36.82.7:4910 , piece_id=ZJYKL4SR6BZP6OYD7QQOZADAR7GDVKVXV4GERU6FQE52RXZGCGOA, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:31.892857589 (61h18m41.615420639s) [upload=58786]: Next returned nil segment Jul 27 00:54:31.892877968 (61h18m41.615441008s) [upload=58786]: Waiting for error group managing segments... Jul 27 00:54:32.223482398 (61h18m41.946045438s) [noise=true , address=45.91.116.6:4974 , piece_id=LR3SRAZFX3NSX7KTZOR643UHIIJSXI4XTBCXYA6W2SFKSEBN7Q2Q, seg_pos={0 0}, upload=58786]: Done uploading piece. err: <nil> Jul 27 00:54:32.237060637 (61h18m41.959623677s) [noise=true , address=45.91.116.4:4939 , piece_id=ZR6IIF5N5GSW6XIMVVRFVAE56M4HNLL5J7SSJE5M5UUFXKZFJSFQ, seg_pos={0 0}, upload=58786]: Done uploading piece. err: <nil> Jul 27 00:54:32.299221539 (61h18m42.021784619s) [noise=true , address=45.91.116.7:4009 , piece_id=4V2UC4QPMJKHBAI7BVHTVPZQFY667XVV3DCXAN3GKL444IHXEVEA, seg_pos={0 0}, upload=58786]: Done uploading piece. err: <nil> Jul 27 00:54:32.325716521 (61h18m42.048279561s) [noise=true , address=45.91.116.6:4973 , piece_id=VKQD7SENX3EDE34BXHGL4Z7MTET2OWSFLTOZ4PPYTDD2QVUHZEJQ, seg_pos={0 0}, upload=58786]: Done uploading piece. err: <nil> Jul 27 00:54:32.325884473 (61h18m42.048447503s) [noise=true , address=45.91.116.12:4227 , piece_id=J2OI4W4CT2FLAUB7WOGOAMTRWRS4PB5AQXLF4XFZNNO2J6LL46SA, seg_pos={0 0}, upload=58786]: Done uploading piece. err: <nil> Jul 27 00:54:32.372983466 (61h18m42.095546506s) [noise=true , address=45.91.116.5:4299 , piece_id=5K6GYVCVQ46VZRUMNZ3PEITFO66VABEGLAV7NZCV5M36QVUVMFCA, seg_pos={0 0}, upload=58786]: Done uploading piece. err: <nil> Jul 27 00:54:32.379844103 (61h18m42.102407133s) [noise=true , address=45.91.116.7:4010 , piece_id=EKECSPS64QHNIO3EIRAVAT35KA5TCPBMDBLT5TSB3LZWKP6KZZHA, seg_pos={0 0}, upload=58786]: Done uploading piece. err: <nil> Jul 27 00:54:32.384666850 (61h18m42.10722989s) [noise=true , address=77.90.31.14:4662 , piece_id=VBVLHQKNOIU25REDYLILDDF3XRZFFU5W5IY4I7YMUE2WTZXLBHEQ, seg_pos={0 0}, upload=58786]: Done uploading piece. err: <nil> Jul 27 00:54:32.472371113 (61h18m42.194934143s) [noise=true , address=77.90.31.16:4735 , piece_id=N32CSF74OC4RGUPMJST5IV5P3FI6RBEUHKXCPXVNUFNHHSN5ZH4A, seg_pos={0 0}, upload=58786]: Done uploading piece. err: <nil> Jul 27 00:54:33.667049984 (61h18m43.389613024s) [noise=true , address=77.90.31.19:4843 , piece_id=KHZD4RVLBISBTIER5RAE2UCFCEFYYWCPMMXLSKIRUTIQTMLODQXQ, seg_pos={0 0}, upload=58786]: Done uploading piece. err: <nil> Jul 27 00:54:34.213712174 (61h18m43.936275214s) [noise=true , address=77.90.31.12:4592 , piece_id=EA7S4T5VCRCJBRETDY6FJ43SOP7NWEVMR2F6L64J3TB6TNOOCKAA, seg_pos={0 0}, upload=58786]: Done uploading piece. err: <nil> Jul 27 00:54:42.252927986 (61h18m51.975491026s) [noise=true , address=77.90.31.10:4520 , piece_id=4GRJDJRCNJPKJ5VV23VRZRNBLAOCJEHFVW534O3M3TQ4SRSP2FEQ, seg_pos={0 0}, upload=58786]: Done uploading piece. err: <nil> Jul 27 00:54:59.328639826 (61h19m9.051202866s) [noise=true , address=104.36.82.5:4838 , piece_id=JNEB4FUOK534MWNRW4DB5VJPPCUIBDA6FIX2YXWSRXNFQ6GDACHA, seg_pos={0 0}, upload=58786]: Done uploading piece. err: ecclient: upload failed (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL, address:104.36.82.5:4838): protocol: rpc: dial tcp 104.36.82.5:4838: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:54:59.328713146 (61h19m9.051276186s) [noise=true , address=104.36.82.7:4910 , piece_id=ZJYKL4SR6BZP6OYD7QQOZADAR7GDVKVXV4GERU6FQE52RXZGCGOA, seg_pos={0 0}, upload=58786]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:104.36.82.7:4910): protocol: rpc: dial tcp 104.36.82.7:4910: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:54:59.328897458 (61h19m9.051460488s) [noise=true , address=104.36.82.15:4259 , piece_id=2CYXGFMVV7Z4PMRHEJ23VWSK53BM3C4DRQPNSPOI55MY7FLS2OSQ, seg_pos={0 0}, upload=58786]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:104.36.82.15:4259): protocol: write tcp 188.166.225.170:42398->104.36.82.15:4259: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:54:59.329496955 (61h19m9.052059995s) [noise=true , address=104.36.82.9:4005 , piece_id=3ARJN3YOZQ7Y3YYCSDQOHV5PEWB7WWJSS3WCHLJ52KWKD32CUKDQ, seg_pos={0 0}, upload=58786]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:104.36.82.9:4005): protocol: rpc: dial tcp 104.36.82.9:4005: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:54:59.329528055 (61h19m9.052091095s) [seg_pos={0 0}, upload=58786]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 27 00:54:59.557689493 (61h19m9.280252533s) [upload=58786]: Done uploading segments... Jul 27 00:54:59.557710713 (61h19m9.280273753s) [upload=58786]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.848326128s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 4 out of 60 is failed (threshold=3) [..............X....X..........X...................X.........] duration: 1.874862514s FAIL (2025-07-27 01:26:03 UTC) orbiter warning: digest mismatch: after 792f8e52f0f261131431c48786048ce73833f71a202ca313ecdd5b39aa603e1c != before 4c0418a6860f4eb95c9171181a1c1a28de1d7886deefc8fd7b2ed1a91cded834 Duration: 1.874884145s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 4 out of 60 is failed (threshold=3) [...............X....X..........X...................X........] duration: 1.906638489s FAIL (2025-07-27 01:25:43 UTC) orbiter warning: digest mismatch: after 792f8e52f0f261131431c48786048ce73833f71a202ca313ecdd5b39aa603e1c != before 4c0418a6860f4eb95c9171181a1c1a28de1d7886deefc8fd7b2ed1a91cded834 Duration: 1.906661936s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 5 out of 60 is failed (threshold=3) [....X....X....XXX...........................................] duration: 5.83896128s FAIL (2025-07-27 01:25:36 UTC) orbiter warning: uplink: context deadline exceeded Duration: 5.839002841s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 10 out of 60 is failed (threshold=3) [XXXX.......XXXXXX...........................................] duration: 5.55701225s FAIL (2025-07-27 01:25:33 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== false Jul 27 00:54:57.821101936 (61h19m7.543664977s) [upload=58793]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 00:54:57.821108168 (61h19m7.543671208s) [upload=58793]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 00:54:58.268999381 (61h19m7.991562411s) [seg_pos={0 0}, upload=58793]: Begin upload segment... Jul 27 00:54:58.269047784 (61h19m7.991610824s) [seg_pos={0 0}, upload=58793]: Done begin upload segment. Jul 27 00:54:58.269068714 (61h19m7.991631754s) [noise=true , address=79.127.201.215:4344 , piece_id=BFA6WIJYTUHSVLKQVLXWCBKNYJJB6CXUUVTZMZEESHH4QG3OV4NQ, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.269108290 (61h19m7.99167133s) [noise=true , address=79.127.205.235:4682 , piece_id=4BXTF4IATE3AD46376DXSOM25E3UL75T4DYKSI7YKKAY4XDUGXZQ, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.269416039 (61h19m7.991979079s) [noise=true , address=104.36.82.53:4153 , piece_id=S2YS2I44RPQEKILNXGKZ6JKUWZIU47ZJTQTDDH6IKCKCEL7IM2NA, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.269588800 (61h19m7.99215184s) [noise=true , address=64.49.13.137:4502 , piece_id=COTTEJEOI7INJS3ZDBHS2CLIGMREMHZGHN7MB22TJR2Y4LZMVRRA, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.269666007 (61h19m7.992229048s) [noise=true , address=77.90.31.11:4573 , piece_id=352UN4WWJYSEOQ4VR4CDGD5XWCDY6TXNXZJW6COACYDDPNTGZSOA, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.269938439 (61h19m7.992501479s) [noise=true , address=104.36.82.46:4500 , piece_id=SQPQJFK3QRENUZNGARKNCCVDHDY2ZXIZNVOFZV76AC63YFACQTBQ, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.270186764 (61h19m7.992749794s) [noise=true , address=79.127.219.38:4001 , piece_id=ZKBCPE6IA6NQJDRGDGFQT2NNGIXAH64ZZ27OE2WPUR54XOQQ2Q3Q, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.270456489 (61h19m7.993019529s) [noise=true , address=104.36.82.47:4545 , piece_id=F4LGWDTZ3CMZIZBDGZZ4TGZ6GKFYE3GNW2B6SZEXBBMYYFS6Y3CQ, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.270693122 (61h19m7.993256162s) [noise=true , address=104.36.82.15:4289 , piece_id=MFZHE2Q766VXNQ43QOIPXO33RASSYDVTJ2QLTVR4ACCHMGJLVV2A, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.270924895 (61h19m7.993487935s) [noise=true , address=77.90.31.10:4525 , piece_id=B7APUOSESFZRV2QUHZ6KZLR7VRJTPHNZ3FFWMEPYV6O7I7EQFVDA, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.270973609 (61h19m7.993536649s) [noise=true , address=104.36.82.52:4091 , piece_id=6Z4ZKBKTEMXCSMNIC6277XZMSJPBKTFW5RJ2PC726VVJTALRJ6NA, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.271180806 (61h19m7.993743846s) [noise=true , address=79.127.205.245:4242 , piece_id=MH52WQ77OW4ZDQFVVFICHDWERB3WGGZXONOOA66J4P32ILZAREQQ, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.271367583 (61h19m7.993930613s) [noise=true , address=64.49.13.140:4514 , piece_id=RXOR5I3R3HF6WUJPKES4OQFTPMETGOEFPZB2KTUS4WVSQUIQQTDA, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.271609946 (61h19m7.994172986s) [noise=true , address=156.146.43.225:4448 , piece_id=TAPXLBQNT5ERYAWMWFBHQF4LYXAZJS4XXIAHXT57S3ORLKEX4MQQ, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.271884882 (61h19m7.994447922s) [noise=true , address=104.36.82.49:4005 , piece_id=J6YISXBRJWWSOJLDNE4U72EQFENIS7SPYZM4SCEVJG2SUGDHRNDA, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.271925130 (61h19m7.99448817s) [noise=true , address=79.127.219.47:4072 , piece_id=JBTVVQE3YIPDNKFAWIPDB5UBQZXP7KJJW4OBCXTAZDUQWLODTZWQ, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.272230464 (61h19m7.994793504s) [noise=true , address=77.90.31.16:4751 , piece_id=XC46QZVCMKTVTLQYDPGM4ZRRD4KDIFCJTQMHFDMY633FQG74FWWQ, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.272352857 (61h19m7.994915898s) [noise=true , address=79.127.220.101:4041 , piece_id=DBBBRJX5NPIPW5P72KSEANTJRZEM5MMUJQQTLXDSCYVUQY6IIIXA, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:59.403085511 (61h19m9.125648551s) [upload=58793]: Next returned nil segment Jul 27 00:54:59.403106221 (61h19m9.125669281s) [upload=58793]: Waiting for error group managing segments... Jul 27 00:54:59.699317478 (61h19m9.421880518s) [noise=true , address=64.49.13.137:4502 , piece_id=COTTEJEOI7INJS3ZDBHS2CLIGMREMHZGHN7MB22TJR2Y4LZMVRRA, seg_pos={0 0}, upload=58793]: Done uploading piece. err: <nil> Jul 27 00:54:59.712155049 (61h19m9.434718079s) [noise=true , address=79.127.205.235:4682 , piece_id=4BXTF4IATE3AD46376DXSOM25E3UL75T4DYKSI7YKKAY4XDUGXZQ, seg_pos={0 0}, upload=58793]: Done uploading piece. err: <nil> Jul 27 00:54:59.718874015 (61h19m9.441437055s) [noise=true , address=79.127.205.245:4242 , piece_id=MH52WQ77OW4ZDQFVVFICHDWERB3WGGZXONOOA66J4P32ILZAREQQ, seg_pos={0 0}, upload=58793]: Done uploading piece. err: <nil> Jul 27 00:55:00.219903397 (61h19m9.942466437s) [noise=true , address=77.90.31.16:4751 , piece_id=XC46QZVCMKTVTLQYDPGM4ZRRD4KDIFCJTQMHFDMY633FQG74FWWQ, seg_pos={0 0}, upload=58793]: Done uploading piece. err: <nil> Jul 27 00:55:00.393745135 (61h19m10.116308165s) [noise=true , address=77.90.31.11:4573 , piece_id=352UN4WWJYSEOQ4VR4CDGD5XWCDY6TXNXZJW6COACYDDPNTGZSOA, seg_pos={0 0}, upload=58793]: Done uploading piece. err: <nil> Jul 27 00:55:01.589079921 (61h19m11.311642972s) [noise=true , address=79.127.219.47:4072 , piece_id=JBTVVQE3YIPDNKFAWIPDB5UBQZXP7KJJW4OBCXTAZDUQWLODTZWQ, seg_pos={0 0}, upload=58793]: Done uploading piece. err: <nil> Jul 27 00:55:01.684825540 (61h19m11.40738858s) [noise=true , address=79.127.220.101:4041 , piece_id=DBBBRJX5NPIPW5P72KSEANTJRZEM5MMUJQQTLXDSCYVUQY6IIIXA, seg_pos={0 0}, upload=58793]: Done uploading piece. err: <nil> Jul 27 00:55:02.296389006 (61h19m12.018952046s) [noise=true , address=156.146.43.225:4448 , piece_id=TAPXLBQNT5ERYAWMWFBHQF4LYXAZJS4XXIAHXT57S3ORLKEX4MQQ, seg_pos={0 0}, upload=58793]: Done uploading piece. err: <nil> Jul 27 00:55:03.015602295 (61h19m12.738165335s) [noise=true , address=79.127.219.38:4001 , piece_id=ZKBCPE6IA6NQJDRGDGFQT2NNGIXAH64ZZ27OE2WPUR54XOQQ2Q3Q, seg_pos={0 0}, upload=58793]: Done uploading piece. err: <nil> Jul 27 00:55:03.588380603 (61h19m13.310943643s) [noise=true , address=77.90.31.10:4525 , piece_id=B7APUOSESFZRV2QUHZ6KZLR7VRJTPHNZ3FFWMEPYV6O7I7EQFVDA, seg_pos={0 0}, upload=58793]: Done uploading piece. err: <nil> Jul 27 00:55:07.419707501 (61h19m17.142270581s) [noise=true , address=79.127.201.215:4344 , piece_id=BFA6WIJYTUHSVLKQVLXWCBKNYJJB6CXUUVTZMZEESHH4QG3OV4NQ, seg_pos={0 0}, upload=58793]: Done uploading piece. err: <nil> Jul 27 00:55:10.064731586 (61h19m19.787294626s) [noise=true , address=64.49.13.140:4514 , piece_id=RXOR5I3R3HF6WUJPKES4OQFTPMETGOEFPZB2KTUS4WVSQUIQQTDA, seg_pos={0 0}, upload=58793]: Done uploading piece. err: <nil> Jul 27 00:55:57.170253087 (61h20m6.892816127s) [noise=true , address=104.36.82.53:4153 , piece_id=S2YS2I44RPQEKILNXGKZ6JKUWZIU47ZJTQTDDH6IKCKCEL7IM2NA, seg_pos={0 0}, upload=58793]: Done uploading piece. err: ecclient: upload failed (node:1fPp4DvLe18dZWPmMPFiC8o9jL4uiEFNmPAHHmFEH74ZXNndw3, address:104.36.82.53:4153): protocol: write tcp 188.166.225.170:41520->104.36.82.53:4153: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:55:57.170329522 (61h20m6.892892562s) [noise=true , address=104.36.82.47:4545 , piece_id=F4LGWDTZ3CMZIZBDGZZ4TGZ6GKFYE3GNW2B6SZEXBBMYYFS6Y3CQ, seg_pos={0 0}, upload=58793]: Done uploading piece. err: ecclient: upload failed (node:12euvmhGegnPYufoKKQ1on9FTv2AhuicSVYuWyXtDiY9BLdDMYE, address:104.36.82.47:4545): protocol: rpc: dial tcp 104.36.82.47:4545: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:55:57.170370181 (61h20m6.892933221s) [noise=true , address=104.36.82.52:4091 , piece_id=6Z4ZKBKTEMXCSMNIC6277XZMSJPBKTFW5RJ2PC726VVJTALRJ6NA, seg_pos={0 0}, upload=58793]: Done uploading piece. err: ecclient: upload failed (node:17wSQzrFc2hUimSsNwDxCgDDi4TpHkbCEH38fGKf9B8WqWzxpW, address:104.36.82.52:4091): protocol: rpc: dial tcp 104.36.82.52:4091: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:55:57.170401270 (61h20m6.89296431s) [noise=true , address=104.36.82.46:4500 , piece_id=SQPQJFK3QRENUZNGARKNCCVDHDY2ZXIZNVOFZV76AC63YFACQTBQ, seg_pos={0 0}, upload=58793]: Done uploading piece. err: ecclient: upload failed (node:12B7ox4ehqxJ2mwqeXj8C9N9GuSU5etNC23iT1pJnTgMLGBAGQK, address:104.36.82.46:4500): protocol: rpc: dial tcp 104.36.82.46:4500: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:55:57.170514156 (61h20m6.893077196s) [noise=true , address=104.36.82.49:4005 , piece_id=J6YISXBRJWWSOJLDNE4U72EQFENIS7SPYZM4SCEVJG2SUGDHRNDA, seg_pos={0 0}, upload=58793]: Done uploading piece. err: ecclient: upload failed (node:1AivThSVLAp9zneNK79sic4AqnTQ62yvNTFF36imigarMGqDSW, address:104.36.82.49:4005): protocol: rpc: dial tcp 104.36.82.49:4005: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:55:57.170524104 (61h20m6.893087155s) [noise=true , address=104.36.82.15:4289 , piece_id=MFZHE2Q766VXNQ43QOIPXO33RASSYDVTJ2QLTVR4ACCHMGJLVV2A, seg_pos={0 0}, upload=58793]: Done uploading piece. err: ecclient: upload failed (node:126WPF6MtX7YjuKf73rGJrifdXyd78KY4FdL757qDipGNHBW6uY, address:104.36.82.15:4289): protocol: rpc: dial tcp 104.36.82.15:4289: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:55:57.170556958 (61h20m6.893119998s) [seg_pos={0 0}, upload=58793]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Jul 27 00:55:57.400256958 (61h20m7.122819998s) [upload=58793]: Done uploading segments... Jul 27 00:55:57.400281876 (61h20m7.122844916s) [upload=58793]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 5.557044021s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [.X.X..............X.........................................] duration: 5.786937093s FAIL (2025-07-27 01:25:31 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.786955879s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 5 out of 60 is failed (threshold=3) [.....X....X....XXX..........................................] duration: 6.484656613s FAIL (2025-07-27 01:25:19 UTC) orbiter warning: uplink: context deadline exceeded Duration: 6.484714694s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 11 out of 60 is failed (threshold=3) [XXXXX.......XXXXXX..........................................] duration: 5.763416061s FAIL (2025-07-27 01:25:07 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== false Jul 27 00:54:57.821101936 (61h19m7.543664977s) [upload=58793]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 00:54:57.821108168 (61h19m7.543671208s) [upload=58793]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 00:54:58.268999381 (61h19m7.991562411s) [seg_pos={0 0}, upload=58793]: Begin upload segment... Jul 27 00:54:58.269047784 (61h19m7.991610824s) [seg_pos={0 0}, upload=58793]: Done begin upload segment. Jul 27 00:54:58.269068714 (61h19m7.991631754s) [noise=true , address=79.127.201.215:4344 , piece_id=BFA6WIJYTUHSVLKQVLXWCBKNYJJB6CXUUVTZMZEESHH4QG3OV4NQ, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.269108290 (61h19m7.99167133s) [noise=true , address=79.127.205.235:4682 , piece_id=4BXTF4IATE3AD46376DXSOM25E3UL75T4DYKSI7YKKAY4XDUGXZQ, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.269416039 (61h19m7.991979079s) [noise=true , address=104.36.82.53:4153 , piece_id=S2YS2I44RPQEKILNXGKZ6JKUWZIU47ZJTQTDDH6IKCKCEL7IM2NA, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.269588800 (61h19m7.99215184s) [noise=true , address=64.49.13.137:4502 , piece_id=COTTEJEOI7INJS3ZDBHS2CLIGMREMHZGHN7MB22TJR2Y4LZMVRRA, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.269666007 (61h19m7.992229048s) [noise=true , address=77.90.31.11:4573 , piece_id=352UN4WWJYSEOQ4VR4CDGD5XWCDY6TXNXZJW6COACYDDPNTGZSOA, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.269938439 (61h19m7.992501479s) [noise=true , address=104.36.82.46:4500 , piece_id=SQPQJFK3QRENUZNGARKNCCVDHDY2ZXIZNVOFZV76AC63YFACQTBQ, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.270186764 (61h19m7.992749794s) [noise=true , address=79.127.219.38:4001 , piece_id=ZKBCPE6IA6NQJDRGDGFQT2NNGIXAH64ZZ27OE2WPUR54XOQQ2Q3Q, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.270456489 (61h19m7.993019529s) [noise=true , address=104.36.82.47:4545 , piece_id=F4LGWDTZ3CMZIZBDGZZ4TGZ6GKFYE3GNW2B6SZEXBBMYYFS6Y3CQ, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.270693122 (61h19m7.993256162s) [noise=true , address=104.36.82.15:4289 , piece_id=MFZHE2Q766VXNQ43QOIPXO33RASSYDVTJ2QLTVR4ACCHMGJLVV2A, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.270924895 (61h19m7.993487935s) [noise=true , address=77.90.31.10:4525 , piece_id=B7APUOSESFZRV2QUHZ6KZLR7VRJTPHNZ3FFWMEPYV6O7I7EQFVDA, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.270973609 (61h19m7.993536649s) [noise=true , address=104.36.82.52:4091 , piece_id=6Z4ZKBKTEMXCSMNIC6277XZMSJPBKTFW5RJ2PC726VVJTALRJ6NA, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.271180806 (61h19m7.993743846s) [noise=true , address=79.127.205.245:4242 , piece_id=MH52WQ77OW4ZDQFVVFICHDWERB3WGGZXONOOA66J4P32ILZAREQQ, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.271367583 (61h19m7.993930613s) [noise=true , address=64.49.13.140:4514 , piece_id=RXOR5I3R3HF6WUJPKES4OQFTPMETGOEFPZB2KTUS4WVSQUIQQTDA, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.271609946 (61h19m7.994172986s) [noise=true , address=156.146.43.225:4448 , piece_id=TAPXLBQNT5ERYAWMWFBHQF4LYXAZJS4XXIAHXT57S3ORLKEX4MQQ, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.271884882 (61h19m7.994447922s) [noise=true , address=104.36.82.49:4005 , piece_id=J6YISXBRJWWSOJLDNE4U72EQFENIS7SPYZM4SCEVJG2SUGDHRNDA, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.271925130 (61h19m7.99448817s) [noise=true , address=79.127.219.47:4072 , piece_id=JBTVVQE3YIPDNKFAWIPDB5UBQZXP7KJJW4OBCXTAZDUQWLODTZWQ, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.272230464 (61h19m7.994793504s) [noise=true , address=77.90.31.16:4751 , piece_id=XC46QZVCMKTVTLQYDPGM4ZRRD4KDIFCJTQMHFDMY633FQG74FWWQ, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.272352857 (61h19m7.994915898s) [noise=true , address=79.127.220.101:4041 , piece_id=DBBBRJX5NPIPW5P72KSEANTJRZEM5MMUJQQTLXDSCYVUQY6IIIXA, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:59.403085511 (61h19m9.125648551s) [upload=58793]: Next returned nil segment Jul 27 00:54:59.403106221 (61h19m9.125669281s) [upload=58793]: Waiting for error group managing segments... Jul 27 00:54:59.699317478 (61h19m9.421880518s) [noise=true , address=64.49.13.137:4502 , piece_id=COTTEJEOI7INJS3ZDBHS2CLIGMREMHZGHN7MB22TJR2Y4LZMVRRA, seg_pos={0 0}, upload=58793]: Done uploading piece. err: <nil> Jul 27 00:54:59.712155049 (61h19m9.434718079s) [noise=true , address=79.127.205.235:4682 , piece_id=4BXTF4IATE3AD46376DXSOM25E3UL75T4DYKSI7YKKAY4XDUGXZQ, seg_pos={0 0}, upload=58793]: Done uploading piece. err: <nil> Jul 27 00:54:59.718874015 (61h19m9.441437055s) [noise=true , address=79.127.205.245:4242 , piece_id=MH52WQ77OW4ZDQFVVFICHDWERB3WGGZXONOOA66J4P32ILZAREQQ, seg_pos={0 0}, upload=58793]: Done uploading piece. err: <nil> Jul 27 00:55:00.219903397 (61h19m9.942466437s) [noise=true , address=77.90.31.16:4751 , piece_id=XC46QZVCMKTVTLQYDPGM4ZRRD4KDIFCJTQMHFDMY633FQG74FWWQ, seg_pos={0 0}, upload=58793]: Done uploading piece. err: <nil> Jul 27 00:55:00.393745135 (61h19m10.116308165s) [noise=true , address=77.90.31.11:4573 , piece_id=352UN4WWJYSEOQ4VR4CDGD5XWCDY6TXNXZJW6COACYDDPNTGZSOA, seg_pos={0 0}, upload=58793]: Done uploading piece. err: <nil> Jul 27 00:55:01.589079921 (61h19m11.311642972s) [noise=true , address=79.127.219.47:4072 , piece_id=JBTVVQE3YIPDNKFAWIPDB5UBQZXP7KJJW4OBCXTAZDUQWLODTZWQ, seg_pos={0 0}, upload=58793]: Done uploading piece. err: <nil> Jul 27 00:55:01.684825540 (61h19m11.40738858s) [noise=true , address=79.127.220.101:4041 , piece_id=DBBBRJX5NPIPW5P72KSEANTJRZEM5MMUJQQTLXDSCYVUQY6IIIXA, seg_pos={0 0}, upload=58793]: Done uploading piece. err: <nil> Jul 27 00:55:02.296389006 (61h19m12.018952046s) [noise=true , address=156.146.43.225:4448 , piece_id=TAPXLBQNT5ERYAWMWFBHQF4LYXAZJS4XXIAHXT57S3ORLKEX4MQQ, seg_pos={0 0}, upload=58793]: Done uploading piece. err: <nil> Jul 27 00:55:03.015602295 (61h19m12.738165335s) [noise=true , address=79.127.219.38:4001 , piece_id=ZKBCPE6IA6NQJDRGDGFQT2NNGIXAH64ZZ27OE2WPUR54XOQQ2Q3Q, seg_pos={0 0}, upload=58793]: Done uploading piece. err: <nil> Jul 27 00:55:03.588380603 (61h19m13.310943643s) [noise=true , address=77.90.31.10:4525 , piece_id=B7APUOSESFZRV2QUHZ6KZLR7VRJTPHNZ3FFWMEPYV6O7I7EQFVDA, seg_pos={0 0}, upload=58793]: Done uploading piece. err: <nil> Jul 27 00:55:07.419707501 (61h19m17.142270581s) [noise=true , address=79.127.201.215:4344 , piece_id=BFA6WIJYTUHSVLKQVLXWCBKNYJJB6CXUUVTZMZEESHH4QG3OV4NQ, seg_pos={0 0}, upload=58793]: Done uploading piece. err: <nil> Jul 27 00:55:10.064731586 (61h19m19.787294626s) [noise=true , address=64.49.13.140:4514 , piece_id=RXOR5I3R3HF6WUJPKES4OQFTPMETGOEFPZB2KTUS4WVSQUIQQTDA, seg_pos={0 0}, upload=58793]: Done uploading piece. err: <nil> Jul 27 00:55:57.170253087 (61h20m6.892816127s) [noise=true , address=104.36.82.53:4153 , piece_id=S2YS2I44RPQEKILNXGKZ6JKUWZIU47ZJTQTDDH6IKCKCEL7IM2NA, seg_pos={0 0}, upload=58793]: Done uploading piece. err: ecclient: upload failed (node:1fPp4DvLe18dZWPmMPFiC8o9jL4uiEFNmPAHHmFEH74ZXNndw3, address:104.36.82.53:4153): protocol: write tcp 188.166.225.170:41520->104.36.82.53:4153: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:55:57.170329522 (61h20m6.892892562s) [noise=true , address=104.36.82.47:4545 , piece_id=F4LGWDTZ3CMZIZBDGZZ4TGZ6GKFYE3GNW2B6SZEXBBMYYFS6Y3CQ, seg_pos={0 0}, upload=58793]: Done uploading piece. err: ecclient: upload failed (node:12euvmhGegnPYufoKKQ1on9FTv2AhuicSVYuWyXtDiY9BLdDMYE, address:104.36.82.47:4545): protocol: rpc: dial tcp 104.36.82.47:4545: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:55:57.170370181 (61h20m6.892933221s) [noise=true , address=104.36.82.52:4091 , piece_id=6Z4ZKBKTEMXCSMNIC6277XZMSJPBKTFW5RJ2PC726VVJTALRJ6NA, seg_pos={0 0}, upload=58793]: Done uploading piece. err: ecclient: upload failed (node:17wSQzrFc2hUimSsNwDxCgDDi4TpHkbCEH38fGKf9B8WqWzxpW, address:104.36.82.52:4091): protocol: rpc: dial tcp 104.36.82.52:4091: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:55:57.170401270 (61h20m6.89296431s) [noise=true , address=104.36.82.46:4500 , piece_id=SQPQJFK3QRENUZNGARKNCCVDHDY2ZXIZNVOFZV76AC63YFACQTBQ, seg_pos={0 0}, upload=58793]: Done uploading piece. err: ecclient: upload failed (node:12B7ox4ehqxJ2mwqeXj8C9N9GuSU5etNC23iT1pJnTgMLGBAGQK, address:104.36.82.46:4500): protocol: rpc: dial tcp 104.36.82.46:4500: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:55:57.170514156 (61h20m6.893077196s) [noise=true , address=104.36.82.49:4005 , piece_id=J6YISXBRJWWSOJLDNE4U72EQFENIS7SPYZM4SCEVJG2SUGDHRNDA, seg_pos={0 0}, upload=58793]: Done uploading piece. err: ecclient: upload failed (node:1AivThSVLAp9zneNK79sic4AqnTQ62yvNTFF36imigarMGqDSW, address:104.36.82.49:4005): protocol: rpc: dial tcp 104.36.82.49:4005: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:55:57.170524104 (61h20m6.893087155s) [noise=true , address=104.36.82.15:4289 , piece_id=MFZHE2Q766VXNQ43QOIPXO33RASSYDVTJ2QLTVR4ACCHMGJLVV2A, seg_pos={0 0}, upload=58793]: Done uploading piece. err: ecclient: upload failed (node:126WPF6MtX7YjuKf73rGJrifdXyd78KY4FdL757qDipGNHBW6uY, address:104.36.82.15:4289): protocol: rpc: dial tcp 104.36.82.15:4289: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:55:57.170556958 (61h20m6.893119998s) [seg_pos={0 0}, upload=58793]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Jul 27 00:55:57.400256958 (61h20m7.122819998s) [upload=58793]: Done uploading segments... Jul 27 00:55:57.400281876 (61h20m7.122844916s) [upload=58793]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 5.76344746s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 6 out of 60 is failed (threshold=3) [X.....X....X....XXX.........................................] duration: 5.860697337s FAIL (2025-07-27 01:25:01 UTC) orbiter warning: uplink: context deadline exceeded Duration: 5.860724269s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 4 out of 60 is failed (threshold=3) [................X....X..........X...................X.......] duration: 1.842041026s FAIL (2025-07-27 01:24:59 UTC) orbiter warning: digest mismatch: after 792f8e52f0f261131431c48786048ce73833f71a202ca313ecdd5b39aa603e1c != before 4c0418a6860f4eb95c9171181a1c1a28de1d7886deefc8fd7b2ed1a91cded834 Duration: 1.842090652s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [XX..X.X.X..........X........................................] duration: 6.226109466s FAIL (2025-07-27 01:24:58 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== 7326QE72JPEXPXFHPPYHSNKIURIMRGIQ, seg_pos={0 0}, upload=58778]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR): protocol: write tcp 188.166.225.170:33562->104.36.82.11:4115: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:54:03.385284649 (61h18m13.107847679s) [seg_pos={0 0}, upload=58778]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 27 00:54:03.385367337 (61h18m13.107930377s) [upload=58778]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 27 00:54:03.385371374 (61h18m13.107934414s) [upload=58778]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 27 00:54:03.688084400 (61h18m13.41064744s) [upload=58778]: Done uploading segments... Jul 27 00:54:03.688101954 (61h18m13.410664994s) [upload=58778]: Upload finished. err: <nil> Jul 27 00:54:29.969393323 (61h18m39.691956364s) [upload=58786]: Starting upload Jul 27 00:54:29.969484158 (61h18m39.692047198s) [upload=58786]: Done starting upload Jul 27 00:54:29.969609477 (61h18m39.692172517s) [upload=58786]: Uploading segments... Jul 27 00:54:29.969791546 (61h18m39.692354576s) [upload=58786]: Got next segment. Inline: false Jul 27 00:54:29.969897859 (61h18m39.6924609s) [upload=58786]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 00:54:29.969903801 (61h18m39.692466841s) [upload=58786]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 00:54:30.202493651 (61h18m39.925056691s) [seg_pos={0 0}, upload=58786]: Begin upload segment... Jul 27 00:54:30.202544298 (61h18m39.925107338s) [seg_pos={0 0}, upload=58786]: Done begin upload segment. Jul 27 00:54:30.202565960 (61h18m39.92512899s) [noise=true , address=104.36.82.15:4259 , piece_id=2CYXGFMVV7Z4PMRHEJ23VWSK53BM3C4DRQPNSPOI55MY7FLS2OSQ, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.202629651 (61h18m39.925192691s) [noise=true , address=45.91.116.5:4299 , piece_id=5K6GYVCVQ46VZRUMNZ3PEITFO66VABEGLAV7NZCV5M36QVUVMFCA, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.202926890 (61h18m39.92548992s) [noise=true , address=45.91.116.7:4010 , piece_id=EKECSPS64QHNIO3EIRAVAT35KA5TCPBMDBLT5TSB3LZWKP6KZZHA, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.203102106 (61h18m39.925665146s) [noise=true , address=77.90.31.16:4735 , piece_id=N32CSF74OC4RGUPMJST5IV5P3FI6RBEUHKXCPXVNUFNHHSN5ZH4A, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.203204351 (61h18m39.925767391s) [noise=true , address=104.36.82.5:4838 , piece_id=JNEB4FUOK534MWNRW4DB5VJPPCUIBDA6FIX2YXWSRXNFQ6GDACHA, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.203445583 (61h18m39.926008623s) [noise=true , address=77.90.31.19:4843 , piece_id=KHZD4RVLBISBTIER5RAE2UCFCEFYYWCPMMXLSKIRUTIQTMLODQXQ, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.203491891 (61h18m39.926054921s) [noise=true , address=77.90.31.10:4520 , piece_id=4GRJDJRCNJPKJ5VV23VRZRNBLAOCJEHFVW534O3M3TQ4SRSP2FEQ, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.203718445 (61h18m39.926281475s) [noise=true , address=45.91.116.6:4974 , piece_id=LR3SRAZFX3NSX7KTZOR643UHIIJSXI4XTBCXYA6W2SFKSEBN7Q2Q, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.204010243 (61h18m39.926573283s) [noise=true , address=45.91.116.12:4227 , piece_id=J2OI4W4CT2FLAUB7WOGOAMTRWRS4PB5AQXLF4XFZNNO2J6LL46SA, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.204257396 (61h18m39.926820436s) [noise=true , address=77.90.31.12:4592 , piece_id=EA7S4T5VCRCJBRETDY6FJ43SOP7NWEVMR2F6L64J3TB6TNOOCKAA, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.204507925 (61h18m39.927070976s) [noise=true , address=45.91.116.4:4939 , piece_id=ZR6IIF5N5GSW6XIMVVRFVAE56M4HNLL5J7SSJE5M5UUFXKZFJSFQ, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.205281365 (61h18m39.927844435s) [noise=true , address=45.91.116.6:4973 , piece_id=VKQD7SENX3EDE34BXHGL4Z7MTET2OWSFLTOZ4PPYTDD2QVUHZEJQ, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.205688314 (61h18m39.928251354s) [noise=true , address=77.90.31.14:4662 , piece_id=VBVLHQKNOIU25REDYLILDDF3XRZFFU5W5IY4I7YMUE2WTZXLBHEQ, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.205981905 (61h18m39.928544945s) [noise=true , address=45.91.116.7:4009 , piece_id=4V2UC4QPMJKHBAI7BVHTVPZQFY667XVV3DCXAN3GKL444IHXEVEA, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.206218718 (61h18m39.928781759s) [noise=true , address=104.36.82.9:4005 , piece_id=3ARJN3YOZQ7Y3YYCSDQOHV5PEWB7WWJSS3WCHLJ52KWKD32CUKDQ, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.207082942 (61h18m39.929645982s) [noise=true , address=104.36.82.7:4910 , piece_id=ZJYKL4SR6BZP6OYD7QQOZADAR7GDVKVXV4GERU6FQE52RXZGCGOA, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:31.892857589 (61h18m41.615420639s) [upload=58786]: Next returned nil segment Jul 27 00:54:31.892877968 (61h18m41.615441008s) [upload=58786]: Waiting for error group managing segments... Jul 27 00:54:32.223482398 (61h18m41.946045438s) [noise=true , address=45.91.116.6:4974 , piece_id=LR3SRAZFX3NSX7KTZOR643UHIIJSXI4XTBCXYA6W2SFKSEBN7Q2Q, seg_pos={0 0}, upload=58786]: Done uploading piece. err: <nil> Jul 27 00:54:32.237060637 (61h18m41.959623677s) [noise=true , address=45.91.116.4:4939 , piece_id=ZR6IIF5N5GSW6XIMVVRFVAE56M4HNLL5J7SSJE5M5UUFXKZFJSFQ, seg_pos={0 0}, upload=58786]: Done uploading piece. err: <nil> Jul 27 00:54:32.299221539 (61h18m42.021784619s) [noise=true , address=45.91.116.7:4009 , piece_id=4V2UC4QPMJKHBAI7BVHTVPZQFY667XVV3DCXAN3GKL444IHXEVEA, seg_pos={0 0}, upload=58786]: Done uploading piece. err: <nil> Jul 27 00:54:32.325716521 (61h18m42.048279561s) [noise=true , address=45.91.116.6:4973 , piece_id=VKQD7SENX3EDE34BXHGL4Z7MTET2OWSFLTOZ4PPYTDD2QVUHZEJQ, seg_pos={0 0}, upload=58786]: Done uploading piece. err: <nil> Jul 27 00:54:32.325884473 (61h18m42.048447503s) [noise=true , address=45.91.116.12:4227 , piece_id=J2OI4W4CT2FLAUB7WOGOAMTRWRS4PB5AQXLF4XFZNNO2J6LL46SA, seg_pos={0 0}, upload=58786]: Done uploading piece. err: <nil> Jul 27 00:54:32.372983466 (61h18m42.095546506s) [noise=true , address=45.91.116.5:4299 , piece_id=5K6GYVCVQ46VZRUMNZ3PEITFO66VABEGLAV7NZCV5M36QVUVMFCA, seg_pos={0 0}, upload=58786]: Done uploading piece. err: <nil> Jul 27 00:54:32.379844103 (61h18m42.102407133s) [noise=true , address=45.91.116.7:4010 , piece_id=EKECSPS64QHNIO3EIRAVAT35KA5TCPBMDBLT5TSB3LZWKP6KZZHA, seg_pos={0 0}, upload=58786]: Done uploading piece. err: <nil> Jul 27 00:54:32.384666850 (61h18m42.10722989s) [noise=true , address=77.90.31.14:4662 , piece_id=VBVLHQKNOIU25REDYLILDDF3XRZFFU5W5IY4I7YMUE2WTZXLBHEQ, seg_pos={0 0}, upload=58786]: Done uploading piece. err: <nil> Jul 27 00:54:32.472371113 (61h18m42.194934143s) [noise=true , address=77.90.31.16:4735 , piece_id=N32CSF74OC4RGUPMJST5IV5P3FI6RBEUHKXCPXVNUFNHHSN5ZH4A, seg_pos={0 0}, upload=58786]: Done uploading piece. err: <nil> Jul 27 00:54:33.667049984 (61h18m43.389613024s) [noise=true , address=77.90.31.19:4843 , piece_id=KHZD4RVLBISBTIER5RAE2UCFCEFYYWCPMMXLSKIRUTIQTMLODQXQ, seg_pos={0 0}, upload=58786]: Done uploading piece. err: <nil> Jul 27 00:54:34.213712174 (61h18m43.936275214s) [noise=true , address=77.90.31.12:4592 , piece_id=EA7S4T5VCRCJBRETDY6FJ43SOP7NWEVMR2F6L64J3TB6TNOOCKAA, seg_pos={0 0}, upload=58786]: Done uploading piece. err: <nil> Jul 27 00:54:42.252927986 (61h18m51.975491026s) [noise=true , address=77.90.31.10:4520 , piece_id=4GRJDJRCNJPKJ5VV23VRZRNBLAOCJEHFVW534O3M3TQ4SRSP2FEQ, seg_pos={0 0}, upload=58786]: Done uploading piece. err: <nil> Jul 27 00:54:59.328639826 (61h19m9.051202866s) [noise=true , address=104.36.82.5:4838 , piece_id=JNEB4FUOK534MWNRW4DB5VJPPCUIBDA6FIX2YXWSRXNFQ6GDACHA, seg_pos={0 0}, upload=58786]: Done uploading piece. err: ecclient: upload failed (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL, address:104.36.82.5:4838): protocol: rpc: dial tcp 104.36.82.5:4838: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:54:59.328713146 (61h19m9.051276186s) [noise=true , address=104.36.82.7:4910 , piece_id=ZJYKL4SR6BZP6OYD7QQOZADAR7GDVKVXV4GERU6FQE52RXZGCGOA, seg_pos={0 0}, upload=58786]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:104.36.82.7:4910): protocol: rpc: dial tcp 104.36.82.7:4910: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:54:59.328897458 (61h19m9.051460488s) [noise=true , address=104.36.82.15:4259 , piece_id=2CYXGFMVV7Z4PMRHEJ23VWSK53BM3C4DRQPNSPOI55MY7FLS2OSQ, seg_pos={0 0}, upload=58786]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:104.36.82.15:4259): protocol: write tcp 188.166.225.170:42398->104.36.82.15:4259: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:54:59.329496955 (61h19m9.052059995s) [noise=true , address=104.36.82.9:4005 , piece_id=3ARJN3YOZQ7Y3YYCSDQOHV5PEWB7WWJSS3WCHLJ52KWKD32CUKDQ, seg_pos={0 0}, upload=58786]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:104.36.82.9:4005): protocol: rpc: dial tcp 104.36.82.9:4005: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:54:59.329528055 (61h19m9.052091095s) [seg_pos={0 0}, upload=58786]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 27 00:54:59.557689493 (61h19m9.280252533s) [upload=58786]: Done uploading segments... Jul 27 00:54:59.557710713 (61h19m9.280273753s) [upload=58786]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 6.226133663s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [XXX..X.X.X..........X.......................................] duration: 10.082613995s FAIL (2025-07-27 01:24:42 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== 7326QE72JPEXPXFHPPYHSNKIURIMRGIQ, seg_pos={0 0}, upload=58778]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR): protocol: write tcp 188.166.225.170:33562->104.36.82.11:4115: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:54:03.385284649 (61h18m13.107847679s) [seg_pos={0 0}, upload=58778]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 27 00:54:03.385367337 (61h18m13.107930377s) [upload=58778]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 27 00:54:03.385371374 (61h18m13.107934414s) [upload=58778]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 27 00:54:03.688084400 (61h18m13.41064744s) [upload=58778]: Done uploading segments... Jul 27 00:54:03.688101954 (61h18m13.410664994s) [upload=58778]: Upload finished. err: <nil> Jul 27 00:54:29.969393323 (61h18m39.691956364s) [upload=58786]: Starting upload Jul 27 00:54:29.969484158 (61h18m39.692047198s) [upload=58786]: Done starting upload Jul 27 00:54:29.969609477 (61h18m39.692172517s) [upload=58786]: Uploading segments... Jul 27 00:54:29.969791546 (61h18m39.692354576s) [upload=58786]: Got next segment. Inline: false Jul 27 00:54:29.969897859 (61h18m39.6924609s) [upload=58786]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 00:54:29.969903801 (61h18m39.692466841s) [upload=58786]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 00:54:30.202493651 (61h18m39.925056691s) [seg_pos={0 0}, upload=58786]: Begin upload segment... Jul 27 00:54:30.202544298 (61h18m39.925107338s) [seg_pos={0 0}, upload=58786]: Done begin upload segment. Jul 27 00:54:30.202565960 (61h18m39.92512899s) [noise=true , address=104.36.82.15:4259 , piece_id=2CYXGFMVV7Z4PMRHEJ23VWSK53BM3C4DRQPNSPOI55MY7FLS2OSQ, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.202629651 (61h18m39.925192691s) [noise=true , address=45.91.116.5:4299 , piece_id=5K6GYVCVQ46VZRUMNZ3PEITFO66VABEGLAV7NZCV5M36QVUVMFCA, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.202926890 (61h18m39.92548992s) [noise=true , address=45.91.116.7:4010 , piece_id=EKECSPS64QHNIO3EIRAVAT35KA5TCPBMDBLT5TSB3LZWKP6KZZHA, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.203102106 (61h18m39.925665146s) [noise=true , address=77.90.31.16:4735 , piece_id=N32CSF74OC4RGUPMJST5IV5P3FI6RBEUHKXCPXVNUFNHHSN5ZH4A, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.203204351 (61h18m39.925767391s) [noise=true , address=104.36.82.5:4838 , piece_id=JNEB4FUOK534MWNRW4DB5VJPPCUIBDA6FIX2YXWSRXNFQ6GDACHA, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.203445583 (61h18m39.926008623s) [noise=true , address=77.90.31.19:4843 , piece_id=KHZD4RVLBISBTIER5RAE2UCFCEFYYWCPMMXLSKIRUTIQTMLODQXQ, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.203491891 (61h18m39.926054921s) [noise=true , address=77.90.31.10:4520 , piece_id=4GRJDJRCNJPKJ5VV23VRZRNBLAOCJEHFVW534O3M3TQ4SRSP2FEQ, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.203718445 (61h18m39.926281475s) [noise=true , address=45.91.116.6:4974 , piece_id=LR3SRAZFX3NSX7KTZOR643UHIIJSXI4XTBCXYA6W2SFKSEBN7Q2Q, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.204010243 (61h18m39.926573283s) [noise=true , address=45.91.116.12:4227 , piece_id=J2OI4W4CT2FLAUB7WOGOAMTRWRS4PB5AQXLF4XFZNNO2J6LL46SA, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.204257396 (61h18m39.926820436s) [noise=true , address=77.90.31.12:4592 , piece_id=EA7S4T5VCRCJBRETDY6FJ43SOP7NWEVMR2F6L64J3TB6TNOOCKAA, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.204507925 (61h18m39.927070976s) [noise=true , address=45.91.116.4:4939 , piece_id=ZR6IIF5N5GSW6XIMVVRFVAE56M4HNLL5J7SSJE5M5UUFXKZFJSFQ, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.205281365 (61h18m39.927844435s) [noise=true , address=45.91.116.6:4973 , piece_id=VKQD7SENX3EDE34BXHGL4Z7MTET2OWSFLTOZ4PPYTDD2QVUHZEJQ, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.205688314 (61h18m39.928251354s) [noise=true , address=77.90.31.14:4662 , piece_id=VBVLHQKNOIU25REDYLILDDF3XRZFFU5W5IY4I7YMUE2WTZXLBHEQ, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.205981905 (61h18m39.928544945s) [noise=true , address=45.91.116.7:4009 , piece_id=4V2UC4QPMJKHBAI7BVHTVPZQFY667XVV3DCXAN3GKL444IHXEVEA, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.206218718 (61h18m39.928781759s) [noise=true , address=104.36.82.9:4005 , piece_id=3ARJN3YOZQ7Y3YYCSDQOHV5PEWB7WWJSS3WCHLJ52KWKD32CUKDQ, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.207082942 (61h18m39.929645982s) [noise=true , address=104.36.82.7:4910 , piece_id=ZJYKL4SR6BZP6OYD7QQOZADAR7GDVKVXV4GERU6FQE52RXZGCGOA, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:31.892857589 (61h18m41.615420639s) [upload=58786]: Next returned nil segment Jul 27 00:54:31.892877968 (61h18m41.615441008s) [upload=58786]: Waiting for error group managing segments... Jul 27 00:54:32.223482398 (61h18m41.946045438s) [noise=true , address=45.91.116.6:4974 , piece_id=LR3SRAZFX3NSX7KTZOR643UHIIJSXI4XTBCXYA6W2SFKSEBN7Q2Q, seg_pos={0 0}, upload=58786]: Done uploading piece. err: <nil> Jul 27 00:54:32.237060637 (61h18m41.959623677s) [noise=true , address=45.91.116.4:4939 , piece_id=ZR6IIF5N5GSW6XIMVVRFVAE56M4HNLL5J7SSJE5M5UUFXKZFJSFQ, seg_pos={0 0}, upload=58786]: Done uploading piece. err: <nil> Jul 27 00:54:32.299221539 (61h18m42.021784619s) [noise=true , address=45.91.116.7:4009 , piece_id=4V2UC4QPMJKHBAI7BVHTVPZQFY667XVV3DCXAN3GKL444IHXEVEA, seg_pos={0 0}, upload=58786]: Done uploading piece. err: <nil> Jul 27 00:54:32.325716521 (61h18m42.048279561s) [noise=true , address=45.91.116.6:4973 , piece_id=VKQD7SENX3EDE34BXHGL4Z7MTET2OWSFLTOZ4PPYTDD2QVUHZEJQ, seg_pos={0 0}, upload=58786]: Done uploading piece. err: <nil> Jul 27 00:54:32.325884473 (61h18m42.048447503s) [noise=true , address=45.91.116.12:4227 , piece_id=J2OI4W4CT2FLAUB7WOGOAMTRWRS4PB5AQXLF4XFZNNO2J6LL46SA, seg_pos={0 0}, upload=58786]: Done uploading piece. err: <nil> Jul 27 00:54:32.372983466 (61h18m42.095546506s) [noise=true , address=45.91.116.5:4299 , piece_id=5K6GYVCVQ46VZRUMNZ3PEITFO66VABEGLAV7NZCV5M36QVUVMFCA, seg_pos={0 0}, upload=58786]: Done uploading piece. err: <nil> Jul 27 00:54:32.379844103 (61h18m42.102407133s) [noise=true , address=45.91.116.7:4010 , piece_id=EKECSPS64QHNIO3EIRAVAT35KA5TCPBMDBLT5TSB3LZWKP6KZZHA, seg_pos={0 0}, upload=58786]: Done uploading piece. err: <nil> Jul 27 00:54:32.384666850 (61h18m42.10722989s) [noise=true , address=77.90.31.14:4662 , piece_id=VBVLHQKNOIU25REDYLILDDF3XRZFFU5W5IY4I7YMUE2WTZXLBHEQ, seg_pos={0 0}, upload=58786]: Done uploading piece. err: <nil> Jul 27 00:54:32.472371113 (61h18m42.194934143s) [noise=true , address=77.90.31.16:4735 , piece_id=N32CSF74OC4RGUPMJST5IV5P3FI6RBEUHKXCPXVNUFNHHSN5ZH4A, seg_pos={0 0}, upload=58786]: Done uploading piece. err: <nil> Jul 27 00:54:33.667049984 (61h18m43.389613024s) [noise=true , address=77.90.31.19:4843 , piece_id=KHZD4RVLBISBTIER5RAE2UCFCEFYYWCPMMXLSKIRUTIQTMLODQXQ, seg_pos={0 0}, upload=58786]: Done uploading piece. err: <nil> Jul 27 00:54:34.213712174 (61h18m43.936275214s) [noise=true , address=77.90.31.12:4592 , piece_id=EA7S4T5VCRCJBRETDY6FJ43SOP7NWEVMR2F6L64J3TB6TNOOCKAA, seg_pos={0 0}, upload=58786]: Done uploading piece. err: <nil> Jul 27 00:54:42.252927986 (61h18m51.975491026s) [noise=true , address=77.90.31.10:4520 , piece_id=4GRJDJRCNJPKJ5VV23VRZRNBLAOCJEHFVW534O3M3TQ4SRSP2FEQ, seg_pos={0 0}, upload=58786]: Done uploading piece. err: <nil> Jul 27 00:54:59.328639826 (61h19m9.051202866s) [noise=true , address=104.36.82.5:4838 , piece_id=JNEB4FUOK534MWNRW4DB5VJPPCUIBDA6FIX2YXWSRXNFQ6GDACHA, seg_pos={0 0}, upload=58786]: Done uploading piece. err: ecclient: upload failed (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL, address:104.36.82.5:4838): protocol: rpc: dial tcp 104.36.82.5:4838: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:54:59.328713146 (61h19m9.051276186s) [noise=true , address=104.36.82.7:4910 , piece_id=ZJYKL4SR6BZP6OYD7QQOZADAR7GDVKVXV4GERU6FQE52RXZGCGOA, seg_pos={0 0}, upload=58786]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:104.36.82.7:4910): protocol: rpc: dial tcp 104.36.82.7:4910: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:54:59.328897458 (61h19m9.051460488s) [noise=true , address=104.36.82.15:4259 , piece_id=2CYXGFMVV7Z4PMRHEJ23VWSK53BM3C4DRQPNSPOI55MY7FLS2OSQ, seg_pos={0 0}, upload=58786]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:104.36.82.15:4259): protocol: write tcp 188.166.225.170:42398->104.36.82.15:4259: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:54:59.329496955 (61h19m9.052059995s) [noise=true , address=104.36.82.9:4005 , piece_id=3ARJN3YOZQ7Y3YYCSDQOHV5PEWB7WWJSS3WCHLJ52KWKD32CUKDQ, seg_pos={0 0}, upload=58786]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:104.36.82.9:4005): protocol: rpc: dial tcp 104.36.82.9:4005: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:54:59.329528055 (61h19m9.052091095s) [seg_pos={0 0}, upload=58786]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 27 00:54:59.557689493 (61h19m9.280252533s) [upload=58786]: Done uploading segments... Jul 27 00:54:59.557710713 (61h19m9.280273753s) [upload=58786]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 10.082633702s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [..X.X..............X........................................] duration: 5.914899282s FAIL (2025-07-27 01:24:33 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.915068877s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 12 out of 60 is failed (threshold=3) [XXXXXX.......XXXXXX.........................................] duration: 5.819026399s FAIL (2025-07-27 01:24:32 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== false Jul 27 00:54:57.821101936 (61h19m7.543664977s) [upload=58793]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 00:54:57.821108168 (61h19m7.543671208s) [upload=58793]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 00:54:58.268999381 (61h19m7.991562411s) [seg_pos={0 0}, upload=58793]: Begin upload segment... Jul 27 00:54:58.269047784 (61h19m7.991610824s) [seg_pos={0 0}, upload=58793]: Done begin upload segment. Jul 27 00:54:58.269068714 (61h19m7.991631754s) [noise=true , address=79.127.201.215:4344 , piece_id=BFA6WIJYTUHSVLKQVLXWCBKNYJJB6CXUUVTZMZEESHH4QG3OV4NQ, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.269108290 (61h19m7.99167133s) [noise=true , address=79.127.205.235:4682 , piece_id=4BXTF4IATE3AD46376DXSOM25E3UL75T4DYKSI7YKKAY4XDUGXZQ, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.269416039 (61h19m7.991979079s) [noise=true , address=104.36.82.53:4153 , piece_id=S2YS2I44RPQEKILNXGKZ6JKUWZIU47ZJTQTDDH6IKCKCEL7IM2NA, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.269588800 (61h19m7.99215184s) [noise=true , address=64.49.13.137:4502 , piece_id=COTTEJEOI7INJS3ZDBHS2CLIGMREMHZGHN7MB22TJR2Y4LZMVRRA, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.269666007 (61h19m7.992229048s) [noise=true , address=77.90.31.11:4573 , piece_id=352UN4WWJYSEOQ4VR4CDGD5XWCDY6TXNXZJW6COACYDDPNTGZSOA, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.269938439 (61h19m7.992501479s) [noise=true , address=104.36.82.46:4500 , piece_id=SQPQJFK3QRENUZNGARKNCCVDHDY2ZXIZNVOFZV76AC63YFACQTBQ, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.270186764 (61h19m7.992749794s) [noise=true , address=79.127.219.38:4001 , piece_id=ZKBCPE6IA6NQJDRGDGFQT2NNGIXAH64ZZ27OE2WPUR54XOQQ2Q3Q, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.270456489 (61h19m7.993019529s) [noise=true , address=104.36.82.47:4545 , piece_id=F4LGWDTZ3CMZIZBDGZZ4TGZ6GKFYE3GNW2B6SZEXBBMYYFS6Y3CQ, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.270693122 (61h19m7.993256162s) [noise=true , address=104.36.82.15:4289 , piece_id=MFZHE2Q766VXNQ43QOIPXO33RASSYDVTJ2QLTVR4ACCHMGJLVV2A, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.270924895 (61h19m7.993487935s) [noise=true , address=77.90.31.10:4525 , piece_id=B7APUOSESFZRV2QUHZ6KZLR7VRJTPHNZ3FFWMEPYV6O7I7EQFVDA, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.270973609 (61h19m7.993536649s) [noise=true , address=104.36.82.52:4091 , piece_id=6Z4ZKBKTEMXCSMNIC6277XZMSJPBKTFW5RJ2PC726VVJTALRJ6NA, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.271180806 (61h19m7.993743846s) [noise=true , address=79.127.205.245:4242 , piece_id=MH52WQ77OW4ZDQFVVFICHDWERB3WGGZXONOOA66J4P32ILZAREQQ, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.271367583 (61h19m7.993930613s) [noise=true , address=64.49.13.140:4514 , piece_id=RXOR5I3R3HF6WUJPKES4OQFTPMETGOEFPZB2KTUS4WVSQUIQQTDA, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.271609946 (61h19m7.994172986s) [noise=true , address=156.146.43.225:4448 , piece_id=TAPXLBQNT5ERYAWMWFBHQF4LYXAZJS4XXIAHXT57S3ORLKEX4MQQ, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.271884882 (61h19m7.994447922s) [noise=true , address=104.36.82.49:4005 , piece_id=J6YISXBRJWWSOJLDNE4U72EQFENIS7SPYZM4SCEVJG2SUGDHRNDA, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.271925130 (61h19m7.99448817s) [noise=true , address=79.127.219.47:4072 , piece_id=JBTVVQE3YIPDNKFAWIPDB5UBQZXP7KJJW4OBCXTAZDUQWLODTZWQ, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.272230464 (61h19m7.994793504s) [noise=true , address=77.90.31.16:4751 , piece_id=XC46QZVCMKTVTLQYDPGM4ZRRD4KDIFCJTQMHFDMY633FQG74FWWQ, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.272352857 (61h19m7.994915898s) [noise=true , address=79.127.220.101:4041 , piece_id=DBBBRJX5NPIPW5P72KSEANTJRZEM5MMUJQQTLXDSCYVUQY6IIIXA, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:59.403085511 (61h19m9.125648551s) [upload=58793]: Next returned nil segment Jul 27 00:54:59.403106221 (61h19m9.125669281s) [upload=58793]: Waiting for error group managing segments... Jul 27 00:54:59.699317478 (61h19m9.421880518s) [noise=true , address=64.49.13.137:4502 , piece_id=COTTEJEOI7INJS3ZDBHS2CLIGMREMHZGHN7MB22TJR2Y4LZMVRRA, seg_pos={0 0}, upload=58793]: Done uploading piece. err: <nil> Jul 27 00:54:59.712155049 (61h19m9.434718079s) [noise=true , address=79.127.205.235:4682 , piece_id=4BXTF4IATE3AD46376DXSOM25E3UL75T4DYKSI7YKKAY4XDUGXZQ, seg_pos={0 0}, upload=58793]: Done uploading piece. err: <nil> Jul 27 00:54:59.718874015 (61h19m9.441437055s) [noise=true , address=79.127.205.245:4242 , piece_id=MH52WQ77OW4ZDQFVVFICHDWERB3WGGZXONOOA66J4P32ILZAREQQ, seg_pos={0 0}, upload=58793]: Done uploading piece. err: <nil> Jul 27 00:55:00.219903397 (61h19m9.942466437s) [noise=true , address=77.90.31.16:4751 , piece_id=XC46QZVCMKTVTLQYDPGM4ZRRD4KDIFCJTQMHFDMY633FQG74FWWQ, seg_pos={0 0}, upload=58793]: Done uploading piece. err: <nil> Jul 27 00:55:00.393745135 (61h19m10.116308165s) [noise=true , address=77.90.31.11:4573 , piece_id=352UN4WWJYSEOQ4VR4CDGD5XWCDY6TXNXZJW6COACYDDPNTGZSOA, seg_pos={0 0}, upload=58793]: Done uploading piece. err: <nil> Jul 27 00:55:01.589079921 (61h19m11.311642972s) [noise=true , address=79.127.219.47:4072 , piece_id=JBTVVQE3YIPDNKFAWIPDB5UBQZXP7KJJW4OBCXTAZDUQWLODTZWQ, seg_pos={0 0}, upload=58793]: Done uploading piece. err: <nil> Jul 27 00:55:01.684825540 (61h19m11.40738858s) [noise=true , address=79.127.220.101:4041 , piece_id=DBBBRJX5NPIPW5P72KSEANTJRZEM5MMUJQQTLXDSCYVUQY6IIIXA, seg_pos={0 0}, upload=58793]: Done uploading piece. err: <nil> Jul 27 00:55:02.296389006 (61h19m12.018952046s) [noise=true , address=156.146.43.225:4448 , piece_id=TAPXLBQNT5ERYAWMWFBHQF4LYXAZJS4XXIAHXT57S3ORLKEX4MQQ, seg_pos={0 0}, upload=58793]: Done uploading piece. err: <nil> Jul 27 00:55:03.015602295 (61h19m12.738165335s) [noise=true , address=79.127.219.38:4001 , piece_id=ZKBCPE6IA6NQJDRGDGFQT2NNGIXAH64ZZ27OE2WPUR54XOQQ2Q3Q, seg_pos={0 0}, upload=58793]: Done uploading piece. err: <nil> Jul 27 00:55:03.588380603 (61h19m13.310943643s) [noise=true , address=77.90.31.10:4525 , piece_id=B7APUOSESFZRV2QUHZ6KZLR7VRJTPHNZ3FFWMEPYV6O7I7EQFVDA, seg_pos={0 0}, upload=58793]: Done uploading piece. err: <nil> Jul 27 00:55:07.419707501 (61h19m17.142270581s) [noise=true , address=79.127.201.215:4344 , piece_id=BFA6WIJYTUHSVLKQVLXWCBKNYJJB6CXUUVTZMZEESHH4QG3OV4NQ, seg_pos={0 0}, upload=58793]: Done uploading piece. err: <nil> Jul 27 00:55:10.064731586 (61h19m19.787294626s) [noise=true , address=64.49.13.140:4514 , piece_id=RXOR5I3R3HF6WUJPKES4OQFTPMETGOEFPZB2KTUS4WVSQUIQQTDA, seg_pos={0 0}, upload=58793]: Done uploading piece. err: <nil> Jul 27 00:55:57.170253087 (61h20m6.892816127s) [noise=true , address=104.36.82.53:4153 , piece_id=S2YS2I44RPQEKILNXGKZ6JKUWZIU47ZJTQTDDH6IKCKCEL7IM2NA, seg_pos={0 0}, upload=58793]: Done uploading piece. err: ecclient: upload failed (node:1fPp4DvLe18dZWPmMPFiC8o9jL4uiEFNmPAHHmFEH74ZXNndw3, address:104.36.82.53:4153): protocol: write tcp 188.166.225.170:41520->104.36.82.53:4153: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:55:57.170329522 (61h20m6.892892562s) [noise=true , address=104.36.82.47:4545 , piece_id=F4LGWDTZ3CMZIZBDGZZ4TGZ6GKFYE3GNW2B6SZEXBBMYYFS6Y3CQ, seg_pos={0 0}, upload=58793]: Done uploading piece. err: ecclient: upload failed (node:12euvmhGegnPYufoKKQ1on9FTv2AhuicSVYuWyXtDiY9BLdDMYE, address:104.36.82.47:4545): protocol: rpc: dial tcp 104.36.82.47:4545: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:55:57.170370181 (61h20m6.892933221s) [noise=true , address=104.36.82.52:4091 , piece_id=6Z4ZKBKTEMXCSMNIC6277XZMSJPBKTFW5RJ2PC726VVJTALRJ6NA, seg_pos={0 0}, upload=58793]: Done uploading piece. err: ecclient: upload failed (node:17wSQzrFc2hUimSsNwDxCgDDi4TpHkbCEH38fGKf9B8WqWzxpW, address:104.36.82.52:4091): protocol: rpc: dial tcp 104.36.82.52:4091: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:55:57.170401270 (61h20m6.89296431s) [noise=true , address=104.36.82.46:4500 , piece_id=SQPQJFK3QRENUZNGARKNCCVDHDY2ZXIZNVOFZV76AC63YFACQTBQ, seg_pos={0 0}, upload=58793]: Done uploading piece. err: ecclient: upload failed (node:12B7ox4ehqxJ2mwqeXj8C9N9GuSU5etNC23iT1pJnTgMLGBAGQK, address:104.36.82.46:4500): protocol: rpc: dial tcp 104.36.82.46:4500: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:55:57.170514156 (61h20m6.893077196s) [noise=true , address=104.36.82.49:4005 , piece_id=J6YISXBRJWWSOJLDNE4U72EQFENIS7SPYZM4SCEVJG2SUGDHRNDA, seg_pos={0 0}, upload=58793]: Done uploading piece. err: ecclient: upload failed (node:1AivThSVLAp9zneNK79sic4AqnTQ62yvNTFF36imigarMGqDSW, address:104.36.82.49:4005): protocol: rpc: dial tcp 104.36.82.49:4005: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:55:57.170524104 (61h20m6.893087155s) [noise=true , address=104.36.82.15:4289 , piece_id=MFZHE2Q766VXNQ43QOIPXO33RASSYDVTJ2QLTVR4ACCHMGJLVV2A, seg_pos={0 0}, upload=58793]: Done uploading piece. err: ecclient: upload failed (node:126WPF6MtX7YjuKf73rGJrifdXyd78KY4FdL757qDipGNHBW6uY, address:104.36.82.15:4289): protocol: rpc: dial tcp 104.36.82.15:4289: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:55:57.170556958 (61h20m6.893119998s) [seg_pos={0 0}, upload=58793]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Jul 27 00:55:57.400256958 (61h20m7.122819998s) [upload=58793]: Done uploading segments... Jul 27 00:55:57.400281876 (61h20m7.122844916s) [upload=58793]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 5.819055794s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 7 out of 60 is failed (threshold=3) [XX.....X....X....XXX........................................] duration: 5.861985791s FAIL (2025-07-27 01:24:17 UTC) orbiter warning: uplink: context deadline exceeded Duration: 5.86201624s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 4 out of 60 is failed (threshold=3) [.................X....X..........X...................X......] duration: 2.004207997s FAIL (2025-07-27 01:24:12 UTC) orbiter warning: digest mismatch: after 792f8e52f0f261131431c48786048ce73833f71a202ca313ecdd5b39aa603e1c != before 4c0418a6860f4eb95c9171181a1c1a28de1d7886deefc8fd7b2ed1a91cded834 Duration: 2.004239317s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 8 out of 60 is failed (threshold=3) [XXX.....X....X....XXX.......................................] duration: 5.975527053s FAIL (2025-07-27 01:23:51 UTC) orbiter warning: uplink: context deadline exceeded Duration: 5.975571408s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [XXXX..X.X.X..........X......................................] duration: 5.856759628s FAIL (2025-07-27 01:23:37 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== 7326QE72JPEXPXFHPPYHSNKIURIMRGIQ, seg_pos={0 0}, upload=58778]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR): protocol: write tcp 188.166.225.170:33562->104.36.82.11:4115: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:54:03.385284649 (61h18m13.107847679s) [seg_pos={0 0}, upload=58778]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 27 00:54:03.385367337 (61h18m13.107930377s) [upload=58778]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 27 00:54:03.385371374 (61h18m13.107934414s) [upload=58778]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 27 00:54:03.688084400 (61h18m13.41064744s) [upload=58778]: Done uploading segments... Jul 27 00:54:03.688101954 (61h18m13.410664994s) [upload=58778]: Upload finished. err: <nil> Jul 27 00:54:29.969393323 (61h18m39.691956364s) [upload=58786]: Starting upload Jul 27 00:54:29.969484158 (61h18m39.692047198s) [upload=58786]: Done starting upload Jul 27 00:54:29.969609477 (61h18m39.692172517s) [upload=58786]: Uploading segments... Jul 27 00:54:29.969791546 (61h18m39.692354576s) [upload=58786]: Got next segment. Inline: false Jul 27 00:54:29.969897859 (61h18m39.6924609s) [upload=58786]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 00:54:29.969903801 (61h18m39.692466841s) [upload=58786]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 00:54:30.202493651 (61h18m39.925056691s) [seg_pos={0 0}, upload=58786]: Begin upload segment... Jul 27 00:54:30.202544298 (61h18m39.925107338s) [seg_pos={0 0}, upload=58786]: Done begin upload segment. Jul 27 00:54:30.202565960 (61h18m39.92512899s) [noise=true , address=104.36.82.15:4259 , piece_id=2CYXGFMVV7Z4PMRHEJ23VWSK53BM3C4DRQPNSPOI55MY7FLS2OSQ, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.202629651 (61h18m39.925192691s) [noise=true , address=45.91.116.5:4299 , piece_id=5K6GYVCVQ46VZRUMNZ3PEITFO66VABEGLAV7NZCV5M36QVUVMFCA, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.202926890 (61h18m39.92548992s) [noise=true , address=45.91.116.7:4010 , piece_id=EKECSPS64QHNIO3EIRAVAT35KA5TCPBMDBLT5TSB3LZWKP6KZZHA, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.203102106 (61h18m39.925665146s) [noise=true , address=77.90.31.16:4735 , piece_id=N32CSF74OC4RGUPMJST5IV5P3FI6RBEUHKXCPXVNUFNHHSN5ZH4A, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.203204351 (61h18m39.925767391s) [noise=true , address=104.36.82.5:4838 , piece_id=JNEB4FUOK534MWNRW4DB5VJPPCUIBDA6FIX2YXWSRXNFQ6GDACHA, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.203445583 (61h18m39.926008623s) [noise=true , address=77.90.31.19:4843 , piece_id=KHZD4RVLBISBTIER5RAE2UCFCEFYYWCPMMXLSKIRUTIQTMLODQXQ, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.203491891 (61h18m39.926054921s) [noise=true , address=77.90.31.10:4520 , piece_id=4GRJDJRCNJPKJ5VV23VRZRNBLAOCJEHFVW534O3M3TQ4SRSP2FEQ, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.203718445 (61h18m39.926281475s) [noise=true , address=45.91.116.6:4974 , piece_id=LR3SRAZFX3NSX7KTZOR643UHIIJSXI4XTBCXYA6W2SFKSEBN7Q2Q, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.204010243 (61h18m39.926573283s) [noise=true , address=45.91.116.12:4227 , piece_id=J2OI4W4CT2FLAUB7WOGOAMTRWRS4PB5AQXLF4XFZNNO2J6LL46SA, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.204257396 (61h18m39.926820436s) [noise=true , address=77.90.31.12:4592 , piece_id=EA7S4T5VCRCJBRETDY6FJ43SOP7NWEVMR2F6L64J3TB6TNOOCKAA, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.204507925 (61h18m39.927070976s) [noise=true , address=45.91.116.4:4939 , piece_id=ZR6IIF5N5GSW6XIMVVRFVAE56M4HNLL5J7SSJE5M5UUFXKZFJSFQ, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.205281365 (61h18m39.927844435s) [noise=true , address=45.91.116.6:4973 , piece_id=VKQD7SENX3EDE34BXHGL4Z7MTET2OWSFLTOZ4PPYTDD2QVUHZEJQ, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.205688314 (61h18m39.928251354s) [noise=true , address=77.90.31.14:4662 , piece_id=VBVLHQKNOIU25REDYLILDDF3XRZFFU5W5IY4I7YMUE2WTZXLBHEQ, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.205981905 (61h18m39.928544945s) [noise=true , address=45.91.116.7:4009 , piece_id=4V2UC4QPMJKHBAI7BVHTVPZQFY667XVV3DCXAN3GKL444IHXEVEA, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.206218718 (61h18m39.928781759s) [noise=true , address=104.36.82.9:4005 , piece_id=3ARJN3YOZQ7Y3YYCSDQOHV5PEWB7WWJSS3WCHLJ52KWKD32CUKDQ, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.207082942 (61h18m39.929645982s) [noise=true , address=104.36.82.7:4910 , piece_id=ZJYKL4SR6BZP6OYD7QQOZADAR7GDVKVXV4GERU6FQE52RXZGCGOA, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:31.892857589 (61h18m41.615420639s) [upload=58786]: Next returned nil segment Jul 27 00:54:31.892877968 (61h18m41.615441008s) [upload=58786]: Waiting for error group managing segments... Jul 27 00:54:32.223482398 (61h18m41.946045438s) [noise=true , address=45.91.116.6:4974 , piece_id=LR3SRAZFX3NSX7KTZOR643UHIIJSXI4XTBCXYA6W2SFKSEBN7Q2Q, seg_pos={0 0}, upload=58786]: Done uploading piece. err: <nil> Jul 27 00:54:32.237060637 (61h18m41.959623677s) [noise=true , address=45.91.116.4:4939 , piece_id=ZR6IIF5N5GSW6XIMVVRFVAE56M4HNLL5J7SSJE5M5UUFXKZFJSFQ, seg_pos={0 0}, upload=58786]: Done uploading piece. err: <nil> Jul 27 00:54:32.299221539 (61h18m42.021784619s) [noise=true , address=45.91.116.7:4009 , piece_id=4V2UC4QPMJKHBAI7BVHTVPZQFY667XVV3DCXAN3GKL444IHXEVEA, seg_pos={0 0}, upload=58786]: Done uploading piece. err: <nil> Jul 27 00:54:32.325716521 (61h18m42.048279561s) [noise=true , address=45.91.116.6:4973 , piece_id=VKQD7SENX3EDE34BXHGL4Z7MTET2OWSFLTOZ4PPYTDD2QVUHZEJQ, seg_pos={0 0}, upload=58786]: Done uploading piece. err: <nil> Jul 27 00:54:32.325884473 (61h18m42.048447503s) [noise=true , address=45.91.116.12:4227 , piece_id=J2OI4W4CT2FLAUB7WOGOAMTRWRS4PB5AQXLF4XFZNNO2J6LL46SA, seg_pos={0 0}, upload=58786]: Done uploading piece. err: <nil> Jul 27 00:54:32.372983466 (61h18m42.095546506s) [noise=true , address=45.91.116.5:4299 , piece_id=5K6GYVCVQ46VZRUMNZ3PEITFO66VABEGLAV7NZCV5M36QVUVMFCA, seg_pos={0 0}, upload=58786]: Done uploading piece. err: <nil> Jul 27 00:54:32.379844103 (61h18m42.102407133s) [noise=true , address=45.91.116.7:4010 , piece_id=EKECSPS64QHNIO3EIRAVAT35KA5TCPBMDBLT5TSB3LZWKP6KZZHA, seg_pos={0 0}, upload=58786]: Done uploading piece. err: <nil> Jul 27 00:54:32.384666850 (61h18m42.10722989s) [noise=true , address=77.90.31.14:4662 , piece_id=VBVLHQKNOIU25REDYLILDDF3XRZFFU5W5IY4I7YMUE2WTZXLBHEQ, seg_pos={0 0}, upload=58786]: Done uploading piece. err: <nil> Jul 27 00:54:32.472371113 (61h18m42.194934143s) [noise=true , address=77.90.31.16:4735 , piece_id=N32CSF74OC4RGUPMJST5IV5P3FI6RBEUHKXCPXVNUFNHHSN5ZH4A, seg_pos={0 0}, upload=58786]: Done uploading piece. err: <nil> Jul 27 00:54:33.667049984 (61h18m43.389613024s) [noise=true , address=77.90.31.19:4843 , piece_id=KHZD4RVLBISBTIER5RAE2UCFCEFYYWCPMMXLSKIRUTIQTMLODQXQ, seg_pos={0 0}, upload=58786]: Done uploading piece. err: <nil> Jul 27 00:54:34.213712174 (61h18m43.936275214s) [noise=true , address=77.90.31.12:4592 , piece_id=EA7S4T5VCRCJBRETDY6FJ43SOP7NWEVMR2F6L64J3TB6TNOOCKAA, seg_pos={0 0}, upload=58786]: Done uploading piece. err: <nil> Jul 27 00:54:42.252927986 (61h18m51.975491026s) [noise=true , address=77.90.31.10:4520 , piece_id=4GRJDJRCNJPKJ5VV23VRZRNBLAOCJEHFVW534O3M3TQ4SRSP2FEQ, seg_pos={0 0}, upload=58786]: Done uploading piece. err: <nil> Jul 27 00:54:59.328639826 (61h19m9.051202866s) [noise=true , address=104.36.82.5:4838 , piece_id=JNEB4FUOK534MWNRW4DB5VJPPCUIBDA6FIX2YXWSRXNFQ6GDACHA, seg_pos={0 0}, upload=58786]: Done uploading piece. err: ecclient: upload failed (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL, address:104.36.82.5:4838): protocol: rpc: dial tcp 104.36.82.5:4838: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:54:59.328713146 (61h19m9.051276186s) [noise=true , address=104.36.82.7:4910 , piece_id=ZJYKL4SR6BZP6OYD7QQOZADAR7GDVKVXV4GERU6FQE52RXZGCGOA, seg_pos={0 0}, upload=58786]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:104.36.82.7:4910): protocol: rpc: dial tcp 104.36.82.7:4910: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:54:59.328897458 (61h19m9.051460488s) [noise=true , address=104.36.82.15:4259 , piece_id=2CYXGFMVV7Z4PMRHEJ23VWSK53BM3C4DRQPNSPOI55MY7FLS2OSQ, seg_pos={0 0}, upload=58786]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:104.36.82.15:4259): protocol: write tcp 188.166.225.170:42398->104.36.82.15:4259: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:54:59.329496955 (61h19m9.052059995s) [noise=true , address=104.36.82.9:4005 , piece_id=3ARJN3YOZQ7Y3YYCSDQOHV5PEWB7WWJSS3WCHLJ52KWKD32CUKDQ, seg_pos={0 0}, upload=58786]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:104.36.82.9:4005): protocol: rpc: dial tcp 104.36.82.9:4005: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:54:59.329528055 (61h19m9.052091095s) [seg_pos={0 0}, upload=58786]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 27 00:54:59.557689493 (61h19m9.280252533s) [upload=58786]: Done uploading segments... Jul 27 00:54:59.557710713 (61h19m9.280273753s) [upload=58786]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.856811256s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 13 out of 60 is failed (threshold=3) [XXXXXXX.......XXXXXX........................................] duration: 5.587594543s FAIL (2025-07-27 01:23:36 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== false Jul 27 00:54:57.821101936 (61h19m7.543664977s) [upload=58793]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 00:54:57.821108168 (61h19m7.543671208s) [upload=58793]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 00:54:58.268999381 (61h19m7.991562411s) [seg_pos={0 0}, upload=58793]: Begin upload segment... Jul 27 00:54:58.269047784 (61h19m7.991610824s) [seg_pos={0 0}, upload=58793]: Done begin upload segment. Jul 27 00:54:58.269068714 (61h19m7.991631754s) [noise=true , address=79.127.201.215:4344 , piece_id=BFA6WIJYTUHSVLKQVLXWCBKNYJJB6CXUUVTZMZEESHH4QG3OV4NQ, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.269108290 (61h19m7.99167133s) [noise=true , address=79.127.205.235:4682 , piece_id=4BXTF4IATE3AD46376DXSOM25E3UL75T4DYKSI7YKKAY4XDUGXZQ, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.269416039 (61h19m7.991979079s) [noise=true , address=104.36.82.53:4153 , piece_id=S2YS2I44RPQEKILNXGKZ6JKUWZIU47ZJTQTDDH6IKCKCEL7IM2NA, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.269588800 (61h19m7.99215184s) [noise=true , address=64.49.13.137:4502 , piece_id=COTTEJEOI7INJS3ZDBHS2CLIGMREMHZGHN7MB22TJR2Y4LZMVRRA, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.269666007 (61h19m7.992229048s) [noise=true , address=77.90.31.11:4573 , piece_id=352UN4WWJYSEOQ4VR4CDGD5XWCDY6TXNXZJW6COACYDDPNTGZSOA, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.269938439 (61h19m7.992501479s) [noise=true , address=104.36.82.46:4500 , piece_id=SQPQJFK3QRENUZNGARKNCCVDHDY2ZXIZNVOFZV76AC63YFACQTBQ, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.270186764 (61h19m7.992749794s) [noise=true , address=79.127.219.38:4001 , piece_id=ZKBCPE6IA6NQJDRGDGFQT2NNGIXAH64ZZ27OE2WPUR54XOQQ2Q3Q, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.270456489 (61h19m7.993019529s) [noise=true , address=104.36.82.47:4545 , piece_id=F4LGWDTZ3CMZIZBDGZZ4TGZ6GKFYE3GNW2B6SZEXBBMYYFS6Y3CQ, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.270693122 (61h19m7.993256162s) [noise=true , address=104.36.82.15:4289 , piece_id=MFZHE2Q766VXNQ43QOIPXO33RASSYDVTJ2QLTVR4ACCHMGJLVV2A, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.270924895 (61h19m7.993487935s) [noise=true , address=77.90.31.10:4525 , piece_id=B7APUOSESFZRV2QUHZ6KZLR7VRJTPHNZ3FFWMEPYV6O7I7EQFVDA, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.270973609 (61h19m7.993536649s) [noise=true , address=104.36.82.52:4091 , piece_id=6Z4ZKBKTEMXCSMNIC6277XZMSJPBKTFW5RJ2PC726VVJTALRJ6NA, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.271180806 (61h19m7.993743846s) [noise=true , address=79.127.205.245:4242 , piece_id=MH52WQ77OW4ZDQFVVFICHDWERB3WGGZXONOOA66J4P32ILZAREQQ, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.271367583 (61h19m7.993930613s) [noise=true , address=64.49.13.140:4514 , piece_id=RXOR5I3R3HF6WUJPKES4OQFTPMETGOEFPZB2KTUS4WVSQUIQQTDA, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.271609946 (61h19m7.994172986s) [noise=true , address=156.146.43.225:4448 , piece_id=TAPXLBQNT5ERYAWMWFBHQF4LYXAZJS4XXIAHXT57S3ORLKEX4MQQ, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.271884882 (61h19m7.994447922s) [noise=true , address=104.36.82.49:4005 , piece_id=J6YISXBRJWWSOJLDNE4U72EQFENIS7SPYZM4SCEVJG2SUGDHRNDA, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.271925130 (61h19m7.99448817s) [noise=true , address=79.127.219.47:4072 , piece_id=JBTVVQE3YIPDNKFAWIPDB5UBQZXP7KJJW4OBCXTAZDUQWLODTZWQ, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.272230464 (61h19m7.994793504s) [noise=true , address=77.90.31.16:4751 , piece_id=XC46QZVCMKTVTLQYDPGM4ZRRD4KDIFCJTQMHFDMY633FQG74FWWQ, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.272352857 (61h19m7.994915898s) [noise=true , address=79.127.220.101:4041 , piece_id=DBBBRJX5NPIPW5P72KSEANTJRZEM5MMUJQQTLXDSCYVUQY6IIIXA, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:59.403085511 (61h19m9.125648551s) [upload=58793]: Next returned nil segment Jul 27 00:54:59.403106221 (61h19m9.125669281s) [upload=58793]: Waiting for error group managing segments... Jul 27 00:54:59.699317478 (61h19m9.421880518s) [noise=true , address=64.49.13.137:4502 , piece_id=COTTEJEOI7INJS3ZDBHS2CLIGMREMHZGHN7MB22TJR2Y4LZMVRRA, seg_pos={0 0}, upload=58793]: Done uploading piece. err: <nil> Jul 27 00:54:59.712155049 (61h19m9.434718079s) [noise=true , address=79.127.205.235:4682 , piece_id=4BXTF4IATE3AD46376DXSOM25E3UL75T4DYKSI7YKKAY4XDUGXZQ, seg_pos={0 0}, upload=58793]: Done uploading piece. err: <nil> Jul 27 00:54:59.718874015 (61h19m9.441437055s) [noise=true , address=79.127.205.245:4242 , piece_id=MH52WQ77OW4ZDQFVVFICHDWERB3WGGZXONOOA66J4P32ILZAREQQ, seg_pos={0 0}, upload=58793]: Done uploading piece. err: <nil> Jul 27 00:55:00.219903397 (61h19m9.942466437s) [noise=true , address=77.90.31.16:4751 , piece_id=XC46QZVCMKTVTLQYDPGM4ZRRD4KDIFCJTQMHFDMY633FQG74FWWQ, seg_pos={0 0}, upload=58793]: Done uploading piece. err: <nil> Jul 27 00:55:00.393745135 (61h19m10.116308165s) [noise=true , address=77.90.31.11:4573 , piece_id=352UN4WWJYSEOQ4VR4CDGD5XWCDY6TXNXZJW6COACYDDPNTGZSOA, seg_pos={0 0}, upload=58793]: Done uploading piece. err: <nil> Jul 27 00:55:01.589079921 (61h19m11.311642972s) [noise=true , address=79.127.219.47:4072 , piece_id=JBTVVQE3YIPDNKFAWIPDB5UBQZXP7KJJW4OBCXTAZDUQWLODTZWQ, seg_pos={0 0}, upload=58793]: Done uploading piece. err: <nil> Jul 27 00:55:01.684825540 (61h19m11.40738858s) [noise=true , address=79.127.220.101:4041 , piece_id=DBBBRJX5NPIPW5P72KSEANTJRZEM5MMUJQQTLXDSCYVUQY6IIIXA, seg_pos={0 0}, upload=58793]: Done uploading piece. err: <nil> Jul 27 00:55:02.296389006 (61h19m12.018952046s) [noise=true , address=156.146.43.225:4448 , piece_id=TAPXLBQNT5ERYAWMWFBHQF4LYXAZJS4XXIAHXT57S3ORLKEX4MQQ, seg_pos={0 0}, upload=58793]: Done uploading piece. err: <nil> Jul 27 00:55:03.015602295 (61h19m12.738165335s) [noise=true , address=79.127.219.38:4001 , piece_id=ZKBCPE6IA6NQJDRGDGFQT2NNGIXAH64ZZ27OE2WPUR54XOQQ2Q3Q, seg_pos={0 0}, upload=58793]: Done uploading piece. err: <nil> Jul 27 00:55:03.588380603 (61h19m13.310943643s) [noise=true , address=77.90.31.10:4525 , piece_id=B7APUOSESFZRV2QUHZ6KZLR7VRJTPHNZ3FFWMEPYV6O7I7EQFVDA, seg_pos={0 0}, upload=58793]: Done uploading piece. err: <nil> Jul 27 00:55:07.419707501 (61h19m17.142270581s) [noise=true , address=79.127.201.215:4344 , piece_id=BFA6WIJYTUHSVLKQVLXWCBKNYJJB6CXUUVTZMZEESHH4QG3OV4NQ, seg_pos={0 0}, upload=58793]: Done uploading piece. err: <nil> Jul 27 00:55:10.064731586 (61h19m19.787294626s) [noise=true , address=64.49.13.140:4514 , piece_id=RXOR5I3R3HF6WUJPKES4OQFTPMETGOEFPZB2KTUS4WVSQUIQQTDA, seg_pos={0 0}, upload=58793]: Done uploading piece. err: <nil> Jul 27 00:55:57.170253087 (61h20m6.892816127s) [noise=true , address=104.36.82.53:4153 , piece_id=S2YS2I44RPQEKILNXGKZ6JKUWZIU47ZJTQTDDH6IKCKCEL7IM2NA, seg_pos={0 0}, upload=58793]: Done uploading piece. err: ecclient: upload failed (node:1fPp4DvLe18dZWPmMPFiC8o9jL4uiEFNmPAHHmFEH74ZXNndw3, address:104.36.82.53:4153): protocol: write tcp 188.166.225.170:41520->104.36.82.53:4153: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:55:57.170329522 (61h20m6.892892562s) [noise=true , address=104.36.82.47:4545 , piece_id=F4LGWDTZ3CMZIZBDGZZ4TGZ6GKFYE3GNW2B6SZEXBBMYYFS6Y3CQ, seg_pos={0 0}, upload=58793]: Done uploading piece. err: ecclient: upload failed (node:12euvmhGegnPYufoKKQ1on9FTv2AhuicSVYuWyXtDiY9BLdDMYE, address:104.36.82.47:4545): protocol: rpc: dial tcp 104.36.82.47:4545: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:55:57.170370181 (61h20m6.892933221s) [noise=true , address=104.36.82.52:4091 , piece_id=6Z4ZKBKTEMXCSMNIC6277XZMSJPBKTFW5RJ2PC726VVJTALRJ6NA, seg_pos={0 0}, upload=58793]: Done uploading piece. err: ecclient: upload failed (node:17wSQzrFc2hUimSsNwDxCgDDi4TpHkbCEH38fGKf9B8WqWzxpW, address:104.36.82.52:4091): protocol: rpc: dial tcp 104.36.82.52:4091: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:55:57.170401270 (61h20m6.89296431s) [noise=true , address=104.36.82.46:4500 , piece_id=SQPQJFK3QRENUZNGARKNCCVDHDY2ZXIZNVOFZV76AC63YFACQTBQ, seg_pos={0 0}, upload=58793]: Done uploading piece. err: ecclient: upload failed (node:12B7ox4ehqxJ2mwqeXj8C9N9GuSU5etNC23iT1pJnTgMLGBAGQK, address:104.36.82.46:4500): protocol: rpc: dial tcp 104.36.82.46:4500: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:55:57.170514156 (61h20m6.893077196s) [noise=true , address=104.36.82.49:4005 , piece_id=J6YISXBRJWWSOJLDNE4U72EQFENIS7SPYZM4SCEVJG2SUGDHRNDA, seg_pos={0 0}, upload=58793]: Done uploading piece. err: ecclient: upload failed (node:1AivThSVLAp9zneNK79sic4AqnTQ62yvNTFF36imigarMGqDSW, address:104.36.82.49:4005): protocol: rpc: dial tcp 104.36.82.49:4005: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:55:57.170524104 (61h20m6.893087155s) [noise=true , address=104.36.82.15:4289 , piece_id=MFZHE2Q766VXNQ43QOIPXO33RASSYDVTJ2QLTVR4ACCHMGJLVV2A, seg_pos={0 0}, upload=58793]: Done uploading piece. err: ecclient: upload failed (node:126WPF6MtX7YjuKf73rGJrifdXyd78KY4FdL757qDipGNHBW6uY, address:104.36.82.15:4289): protocol: rpc: dial tcp 104.36.82.15:4289: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:55:57.170556958 (61h20m6.893119998s) [seg_pos={0 0}, upload=58793]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Jul 27 00:55:57.400256958 (61h20m7.122819998s) [upload=58793]: Done uploading segments... Jul 27 00:55:57.400281876 (61h20m7.122844916s) [upload=58793]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 5.587620624s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [X..X.X..............X.......................................] duration: 5.585014966s FAIL (2025-07-27 01:23:28 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.58503779s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 9 out of 60 is failed (threshold=3) [XXXX.....X....X....XXX......................................] duration: 5.776215831s FAIL (2025-07-27 01:23:03 UTC) orbiter warning: uplink: context deadline exceeded Duration: 5.776241701s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 4 out of 60 is failed (threshold=3) [..................X....X..........X...................X.....] duration: 1.941841983s FAIL (2025-07-27 01:22:59 UTC) orbiter warning: digest mismatch: after 792f8e52f0f261131431c48786048ce73833f71a202ca313ecdd5b39aa603e1c != before 4c0418a6860f4eb95c9171181a1c1a28de1d7886deefc8fd7b2ed1a91cded834 Duration: 1.941878252s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [XXXXX..X.X.X..........X.....................................] duration: 5.688280217s FAIL (2025-07-27 01:22:54 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== 7326QE72JPEXPXFHPPYHSNKIURIMRGIQ, seg_pos={0 0}, upload=58778]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR): protocol: write tcp 188.166.225.170:33562->104.36.82.11:4115: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:54:03.385284649 (61h18m13.107847679s) [seg_pos={0 0}, upload=58778]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 27 00:54:03.385367337 (61h18m13.107930377s) [upload=58778]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 27 00:54:03.385371374 (61h18m13.107934414s) [upload=58778]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 27 00:54:03.688084400 (61h18m13.41064744s) [upload=58778]: Done uploading segments... Jul 27 00:54:03.688101954 (61h18m13.410664994s) [upload=58778]: Upload finished. err: <nil> Jul 27 00:54:29.969393323 (61h18m39.691956364s) [upload=58786]: Starting upload Jul 27 00:54:29.969484158 (61h18m39.692047198s) [upload=58786]: Done starting upload Jul 27 00:54:29.969609477 (61h18m39.692172517s) [upload=58786]: Uploading segments... Jul 27 00:54:29.969791546 (61h18m39.692354576s) [upload=58786]: Got next segment. Inline: false Jul 27 00:54:29.969897859 (61h18m39.6924609s) [upload=58786]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 00:54:29.969903801 (61h18m39.692466841s) [upload=58786]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 00:54:30.202493651 (61h18m39.925056691s) [seg_pos={0 0}, upload=58786]: Begin upload segment... Jul 27 00:54:30.202544298 (61h18m39.925107338s) [seg_pos={0 0}, upload=58786]: Done begin upload segment. Jul 27 00:54:30.202565960 (61h18m39.92512899s) [noise=true , address=104.36.82.15:4259 , piece_id=2CYXGFMVV7Z4PMRHEJ23VWSK53BM3C4DRQPNSPOI55MY7FLS2OSQ, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.202629651 (61h18m39.925192691s) [noise=true , address=45.91.116.5:4299 , piece_id=5K6GYVCVQ46VZRUMNZ3PEITFO66VABEGLAV7NZCV5M36QVUVMFCA, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.202926890 (61h18m39.92548992s) [noise=true , address=45.91.116.7:4010 , piece_id=EKECSPS64QHNIO3EIRAVAT35KA5TCPBMDBLT5TSB3LZWKP6KZZHA, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.203102106 (61h18m39.925665146s) [noise=true , address=77.90.31.16:4735 , piece_id=N32CSF74OC4RGUPMJST5IV5P3FI6RBEUHKXCPXVNUFNHHSN5ZH4A, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.203204351 (61h18m39.925767391s) [noise=true , address=104.36.82.5:4838 , piece_id=JNEB4FUOK534MWNRW4DB5VJPPCUIBDA6FIX2YXWSRXNFQ6GDACHA, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.203445583 (61h18m39.926008623s) [noise=true , address=77.90.31.19:4843 , piece_id=KHZD4RVLBISBTIER5RAE2UCFCEFYYWCPMMXLSKIRUTIQTMLODQXQ, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.203491891 (61h18m39.926054921s) [noise=true , address=77.90.31.10:4520 , piece_id=4GRJDJRCNJPKJ5VV23VRZRNBLAOCJEHFVW534O3M3TQ4SRSP2FEQ, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.203718445 (61h18m39.926281475s) [noise=true , address=45.91.116.6:4974 , piece_id=LR3SRAZFX3NSX7KTZOR643UHIIJSXI4XTBCXYA6W2SFKSEBN7Q2Q, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.204010243 (61h18m39.926573283s) [noise=true , address=45.91.116.12:4227 , piece_id=J2OI4W4CT2FLAUB7WOGOAMTRWRS4PB5AQXLF4XFZNNO2J6LL46SA, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.204257396 (61h18m39.926820436s) [noise=true , address=77.90.31.12:4592 , piece_id=EA7S4T5VCRCJBRETDY6FJ43SOP7NWEVMR2F6L64J3TB6TNOOCKAA, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.204507925 (61h18m39.927070976s) [noise=true , address=45.91.116.4:4939 , piece_id=ZR6IIF5N5GSW6XIMVVRFVAE56M4HNLL5J7SSJE5M5UUFXKZFJSFQ, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.205281365 (61h18m39.927844435s) [noise=true , address=45.91.116.6:4973 , piece_id=VKQD7SENX3EDE34BXHGL4Z7MTET2OWSFLTOZ4PPYTDD2QVUHZEJQ, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.205688314 (61h18m39.928251354s) [noise=true , address=77.90.31.14:4662 , piece_id=VBVLHQKNOIU25REDYLILDDF3XRZFFU5W5IY4I7YMUE2WTZXLBHEQ, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.205981905 (61h18m39.928544945s) [noise=true , address=45.91.116.7:4009 , piece_id=4V2UC4QPMJKHBAI7BVHTVPZQFY667XVV3DCXAN3GKL444IHXEVEA, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.206218718 (61h18m39.928781759s) [noise=true , address=104.36.82.9:4005 , piece_id=3ARJN3YOZQ7Y3YYCSDQOHV5PEWB7WWJSS3WCHLJ52KWKD32CUKDQ, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.207082942 (61h18m39.929645982s) [noise=true , address=104.36.82.7:4910 , piece_id=ZJYKL4SR6BZP6OYD7QQOZADAR7GDVKVXV4GERU6FQE52RXZGCGOA, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:31.892857589 (61h18m41.615420639s) [upload=58786]: Next returned nil segment Jul 27 00:54:31.892877968 (61h18m41.615441008s) [upload=58786]: Waiting for error group managing segments... Jul 27 00:54:32.223482398 (61h18m41.946045438s) [noise=true , address=45.91.116.6:4974 , piece_id=LR3SRAZFX3NSX7KTZOR643UHIIJSXI4XTBCXYA6W2SFKSEBN7Q2Q, seg_pos={0 0}, upload=58786]: Done uploading piece. err: <nil> Jul 27 00:54:32.237060637 (61h18m41.959623677s) [noise=true , address=45.91.116.4:4939 , piece_id=ZR6IIF5N5GSW6XIMVVRFVAE56M4HNLL5J7SSJE5M5UUFXKZFJSFQ, seg_pos={0 0}, upload=58786]: Done uploading piece. err: <nil> Jul 27 00:54:32.299221539 (61h18m42.021784619s) [noise=true , address=45.91.116.7:4009 , piece_id=4V2UC4QPMJKHBAI7BVHTVPZQFY667XVV3DCXAN3GKL444IHXEVEA, seg_pos={0 0}, upload=58786]: Done uploading piece. err: <nil> Jul 27 00:54:32.325716521 (61h18m42.048279561s) [noise=true , address=45.91.116.6:4973 , piece_id=VKQD7SENX3EDE34BXHGL4Z7MTET2OWSFLTOZ4PPYTDD2QVUHZEJQ, seg_pos={0 0}, upload=58786]: Done uploading piece. err: <nil> Jul 27 00:54:32.325884473 (61h18m42.048447503s) [noise=true , address=45.91.116.12:4227 , piece_id=J2OI4W4CT2FLAUB7WOGOAMTRWRS4PB5AQXLF4XFZNNO2J6LL46SA, seg_pos={0 0}, upload=58786]: Done uploading piece. err: <nil> Jul 27 00:54:32.372983466 (61h18m42.095546506s) [noise=true , address=45.91.116.5:4299 , piece_id=5K6GYVCVQ46VZRUMNZ3PEITFO66VABEGLAV7NZCV5M36QVUVMFCA, seg_pos={0 0}, upload=58786]: Done uploading piece. err: <nil> Jul 27 00:54:32.379844103 (61h18m42.102407133s) [noise=true , address=45.91.116.7:4010 , piece_id=EKECSPS64QHNIO3EIRAVAT35KA5TCPBMDBLT5TSB3LZWKP6KZZHA, seg_pos={0 0}, upload=58786]: Done uploading piece. err: <nil> Jul 27 00:54:32.384666850 (61h18m42.10722989s) [noise=true , address=77.90.31.14:4662 , piece_id=VBVLHQKNOIU25REDYLILDDF3XRZFFU5W5IY4I7YMUE2WTZXLBHEQ, seg_pos={0 0}, upload=58786]: Done uploading piece. err: <nil> Jul 27 00:54:32.472371113 (61h18m42.194934143s) [noise=true , address=77.90.31.16:4735 , piece_id=N32CSF74OC4RGUPMJST5IV5P3FI6RBEUHKXCPXVNUFNHHSN5ZH4A, seg_pos={0 0}, upload=58786]: Done uploading piece. err: <nil> Jul 27 00:54:33.667049984 (61h18m43.389613024s) [noise=true , address=77.90.31.19:4843 , piece_id=KHZD4RVLBISBTIER5RAE2UCFCEFYYWCPMMXLSKIRUTIQTMLODQXQ, seg_pos={0 0}, upload=58786]: Done uploading piece. err: <nil> Jul 27 00:54:34.213712174 (61h18m43.936275214s) [noise=true , address=77.90.31.12:4592 , piece_id=EA7S4T5VCRCJBRETDY6FJ43SOP7NWEVMR2F6L64J3TB6TNOOCKAA, seg_pos={0 0}, upload=58786]: Done uploading piece. err: <nil> Jul 27 00:54:42.252927986 (61h18m51.975491026s) [noise=true , address=77.90.31.10:4520 , piece_id=4GRJDJRCNJPKJ5VV23VRZRNBLAOCJEHFVW534O3M3TQ4SRSP2FEQ, seg_pos={0 0}, upload=58786]: Done uploading piece. err: <nil> Jul 27 00:54:59.328639826 (61h19m9.051202866s) [noise=true , address=104.36.82.5:4838 , piece_id=JNEB4FUOK534MWNRW4DB5VJPPCUIBDA6FIX2YXWSRXNFQ6GDACHA, seg_pos={0 0}, upload=58786]: Done uploading piece. err: ecclient: upload failed (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL, address:104.36.82.5:4838): protocol: rpc: dial tcp 104.36.82.5:4838: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:54:59.328713146 (61h19m9.051276186s) [noise=true , address=104.36.82.7:4910 , piece_id=ZJYKL4SR6BZP6OYD7QQOZADAR7GDVKVXV4GERU6FQE52RXZGCGOA, seg_pos={0 0}, upload=58786]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:104.36.82.7:4910): protocol: rpc: dial tcp 104.36.82.7:4910: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:54:59.328897458 (61h19m9.051460488s) [noise=true , address=104.36.82.15:4259 , piece_id=2CYXGFMVV7Z4PMRHEJ23VWSK53BM3C4DRQPNSPOI55MY7FLS2OSQ, seg_pos={0 0}, upload=58786]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:104.36.82.15:4259): protocol: write tcp 188.166.225.170:42398->104.36.82.15:4259: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:54:59.329496955 (61h19m9.052059995s) [noise=true , address=104.36.82.9:4005 , piece_id=3ARJN3YOZQ7Y3YYCSDQOHV5PEWB7WWJSS3WCHLJ52KWKD32CUKDQ, seg_pos={0 0}, upload=58786]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:104.36.82.9:4005): protocol: rpc: dial tcp 104.36.82.9:4005: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:54:59.329528055 (61h19m9.052091095s) [seg_pos={0 0}, upload=58786]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 27 00:54:59.557689493 (61h19m9.280252533s) [upload=58786]: Done uploading segments... Jul 27 00:54:59.557710713 (61h19m9.280273753s) [upload=58786]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.688308012s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 14 out of 60 is failed (threshold=3) [XXXXXXXX.......XXXXXX.......................................] duration: 5.904779643s FAIL (2025-07-27 01:22:25 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== false Jul 27 00:54:57.821101936 (61h19m7.543664977s) [upload=58793]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 00:54:57.821108168 (61h19m7.543671208s) [upload=58793]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 00:54:58.268999381 (61h19m7.991562411s) [seg_pos={0 0}, upload=58793]: Begin upload segment... Jul 27 00:54:58.269047784 (61h19m7.991610824s) [seg_pos={0 0}, upload=58793]: Done begin upload segment. Jul 27 00:54:58.269068714 (61h19m7.991631754s) [noise=true , address=79.127.201.215:4344 , piece_id=BFA6WIJYTUHSVLKQVLXWCBKNYJJB6CXUUVTZMZEESHH4QG3OV4NQ, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.269108290 (61h19m7.99167133s) [noise=true , address=79.127.205.235:4682 , piece_id=4BXTF4IATE3AD46376DXSOM25E3UL75T4DYKSI7YKKAY4XDUGXZQ, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.269416039 (61h19m7.991979079s) [noise=true , address=104.36.82.53:4153 , piece_id=S2YS2I44RPQEKILNXGKZ6JKUWZIU47ZJTQTDDH6IKCKCEL7IM2NA, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.269588800 (61h19m7.99215184s) [noise=true , address=64.49.13.137:4502 , piece_id=COTTEJEOI7INJS3ZDBHS2CLIGMREMHZGHN7MB22TJR2Y4LZMVRRA, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.269666007 (61h19m7.992229048s) [noise=true , address=77.90.31.11:4573 , piece_id=352UN4WWJYSEOQ4VR4CDGD5XWCDY6TXNXZJW6COACYDDPNTGZSOA, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.269938439 (61h19m7.992501479s) [noise=true , address=104.36.82.46:4500 , piece_id=SQPQJFK3QRENUZNGARKNCCVDHDY2ZXIZNVOFZV76AC63YFACQTBQ, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.270186764 (61h19m7.992749794s) [noise=true , address=79.127.219.38:4001 , piece_id=ZKBCPE6IA6NQJDRGDGFQT2NNGIXAH64ZZ27OE2WPUR54XOQQ2Q3Q, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.270456489 (61h19m7.993019529s) [noise=true , address=104.36.82.47:4545 , piece_id=F4LGWDTZ3CMZIZBDGZZ4TGZ6GKFYE3GNW2B6SZEXBBMYYFS6Y3CQ, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.270693122 (61h19m7.993256162s) [noise=true , address=104.36.82.15:4289 , piece_id=MFZHE2Q766VXNQ43QOIPXO33RASSYDVTJ2QLTVR4ACCHMGJLVV2A, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.270924895 (61h19m7.993487935s) [noise=true , address=77.90.31.10:4525 , piece_id=B7APUOSESFZRV2QUHZ6KZLR7VRJTPHNZ3FFWMEPYV6O7I7EQFVDA, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.270973609 (61h19m7.993536649s) [noise=true , address=104.36.82.52:4091 , piece_id=6Z4ZKBKTEMXCSMNIC6277XZMSJPBKTFW5RJ2PC726VVJTALRJ6NA, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.271180806 (61h19m7.993743846s) [noise=true , address=79.127.205.245:4242 , piece_id=MH52WQ77OW4ZDQFVVFICHDWERB3WGGZXONOOA66J4P32ILZAREQQ, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.271367583 (61h19m7.993930613s) [noise=true , address=64.49.13.140:4514 , piece_id=RXOR5I3R3HF6WUJPKES4OQFTPMETGOEFPZB2KTUS4WVSQUIQQTDA, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.271609946 (61h19m7.994172986s) [noise=true , address=156.146.43.225:4448 , piece_id=TAPXLBQNT5ERYAWMWFBHQF4LYXAZJS4XXIAHXT57S3ORLKEX4MQQ, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.271884882 (61h19m7.994447922s) [noise=true , address=104.36.82.49:4005 , piece_id=J6YISXBRJWWSOJLDNE4U72EQFENIS7SPYZM4SCEVJG2SUGDHRNDA, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.271925130 (61h19m7.99448817s) [noise=true , address=79.127.219.47:4072 , piece_id=JBTVVQE3YIPDNKFAWIPDB5UBQZXP7KJJW4OBCXTAZDUQWLODTZWQ, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.272230464 (61h19m7.994793504s) [noise=true , address=77.90.31.16:4751 , piece_id=XC46QZVCMKTVTLQYDPGM4ZRRD4KDIFCJTQMHFDMY633FQG74FWWQ, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.272352857 (61h19m7.994915898s) [noise=true , address=79.127.220.101:4041 , piece_id=DBBBRJX5NPIPW5P72KSEANTJRZEM5MMUJQQTLXDSCYVUQY6IIIXA, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:59.403085511 (61h19m9.125648551s) [upload=58793]: Next returned nil segment Jul 27 00:54:59.403106221 (61h19m9.125669281s) [upload=58793]: Waiting for error group managing segments... Jul 27 00:54:59.699317478 (61h19m9.421880518s) [noise=true , address=64.49.13.137:4502 , piece_id=COTTEJEOI7INJS3ZDBHS2CLIGMREMHZGHN7MB22TJR2Y4LZMVRRA, seg_pos={0 0}, upload=58793]: Done uploading piece. err: <nil> Jul 27 00:54:59.712155049 (61h19m9.434718079s) [noise=true , address=79.127.205.235:4682 , piece_id=4BXTF4IATE3AD46376DXSOM25E3UL75T4DYKSI7YKKAY4XDUGXZQ, seg_pos={0 0}, upload=58793]: Done uploading piece. err: <nil> Jul 27 00:54:59.718874015 (61h19m9.441437055s) [noise=true , address=79.127.205.245:4242 , piece_id=MH52WQ77OW4ZDQFVVFICHDWERB3WGGZXONOOA66J4P32ILZAREQQ, seg_pos={0 0}, upload=58793]: Done uploading piece. err: <nil> Jul 27 00:55:00.219903397 (61h19m9.942466437s) [noise=true , address=77.90.31.16:4751 , piece_id=XC46QZVCMKTVTLQYDPGM4ZRRD4KDIFCJTQMHFDMY633FQG74FWWQ, seg_pos={0 0}, upload=58793]: Done uploading piece. err: <nil> Jul 27 00:55:00.393745135 (61h19m10.116308165s) [noise=true , address=77.90.31.11:4573 , piece_id=352UN4WWJYSEOQ4VR4CDGD5XWCDY6TXNXZJW6COACYDDPNTGZSOA, seg_pos={0 0}, upload=58793]: Done uploading piece. err: <nil> Jul 27 00:55:01.589079921 (61h19m11.311642972s) [noise=true , address=79.127.219.47:4072 , piece_id=JBTVVQE3YIPDNKFAWIPDB5UBQZXP7KJJW4OBCXTAZDUQWLODTZWQ, seg_pos={0 0}, upload=58793]: Done uploading piece. err: <nil> Jul 27 00:55:01.684825540 (61h19m11.40738858s) [noise=true , address=79.127.220.101:4041 , piece_id=DBBBRJX5NPIPW5P72KSEANTJRZEM5MMUJQQTLXDSCYVUQY6IIIXA, seg_pos={0 0}, upload=58793]: Done uploading piece. err: <nil> Jul 27 00:55:02.296389006 (61h19m12.018952046s) [noise=true , address=156.146.43.225:4448 , piece_id=TAPXLBQNT5ERYAWMWFBHQF4LYXAZJS4XXIAHXT57S3ORLKEX4MQQ, seg_pos={0 0}, upload=58793]: Done uploading piece. err: <nil> Jul 27 00:55:03.015602295 (61h19m12.738165335s) [noise=true , address=79.127.219.38:4001 , piece_id=ZKBCPE6IA6NQJDRGDGFQT2NNGIXAH64ZZ27OE2WPUR54XOQQ2Q3Q, seg_pos={0 0}, upload=58793]: Done uploading piece. err: <nil> Jul 27 00:55:03.588380603 (61h19m13.310943643s) [noise=true , address=77.90.31.10:4525 , piece_id=B7APUOSESFZRV2QUHZ6KZLR7VRJTPHNZ3FFWMEPYV6O7I7EQFVDA, seg_pos={0 0}, upload=58793]: Done uploading piece. err: <nil> Jul 27 00:55:07.419707501 (61h19m17.142270581s) [noise=true , address=79.127.201.215:4344 , piece_id=BFA6WIJYTUHSVLKQVLXWCBKNYJJB6CXUUVTZMZEESHH4QG3OV4NQ, seg_pos={0 0}, upload=58793]: Done uploading piece. err: <nil> Jul 27 00:55:10.064731586 (61h19m19.787294626s) [noise=true , address=64.49.13.140:4514 , piece_id=RXOR5I3R3HF6WUJPKES4OQFTPMETGOEFPZB2KTUS4WVSQUIQQTDA, seg_pos={0 0}, upload=58793]: Done uploading piece. err: <nil> Jul 27 00:55:57.170253087 (61h20m6.892816127s) [noise=true , address=104.36.82.53:4153 , piece_id=S2YS2I44RPQEKILNXGKZ6JKUWZIU47ZJTQTDDH6IKCKCEL7IM2NA, seg_pos={0 0}, upload=58793]: Done uploading piece. err: ecclient: upload failed (node:1fPp4DvLe18dZWPmMPFiC8o9jL4uiEFNmPAHHmFEH74ZXNndw3, address:104.36.82.53:4153): protocol: write tcp 188.166.225.170:41520->104.36.82.53:4153: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:55:57.170329522 (61h20m6.892892562s) [noise=true , address=104.36.82.47:4545 , piece_id=F4LGWDTZ3CMZIZBDGZZ4TGZ6GKFYE3GNW2B6SZEXBBMYYFS6Y3CQ, seg_pos={0 0}, upload=58793]: Done uploading piece. err: ecclient: upload failed (node:12euvmhGegnPYufoKKQ1on9FTv2AhuicSVYuWyXtDiY9BLdDMYE, address:104.36.82.47:4545): protocol: rpc: dial tcp 104.36.82.47:4545: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:55:57.170370181 (61h20m6.892933221s) [noise=true , address=104.36.82.52:4091 , piece_id=6Z4ZKBKTEMXCSMNIC6277XZMSJPBKTFW5RJ2PC726VVJTALRJ6NA, seg_pos={0 0}, upload=58793]: Done uploading piece. err: ecclient: upload failed (node:17wSQzrFc2hUimSsNwDxCgDDi4TpHkbCEH38fGKf9B8WqWzxpW, address:104.36.82.52:4091): protocol: rpc: dial tcp 104.36.82.52:4091: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:55:57.170401270 (61h20m6.89296431s) [noise=true , address=104.36.82.46:4500 , piece_id=SQPQJFK3QRENUZNGARKNCCVDHDY2ZXIZNVOFZV76AC63YFACQTBQ, seg_pos={0 0}, upload=58793]: Done uploading piece. err: ecclient: upload failed (node:12B7ox4ehqxJ2mwqeXj8C9N9GuSU5etNC23iT1pJnTgMLGBAGQK, address:104.36.82.46:4500): protocol: rpc: dial tcp 104.36.82.46:4500: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:55:57.170514156 (61h20m6.893077196s) [noise=true , address=104.36.82.49:4005 , piece_id=J6YISXBRJWWSOJLDNE4U72EQFENIS7SPYZM4SCEVJG2SUGDHRNDA, seg_pos={0 0}, upload=58793]: Done uploading piece. err: ecclient: upload failed (node:1AivThSVLAp9zneNK79sic4AqnTQ62yvNTFF36imigarMGqDSW, address:104.36.82.49:4005): protocol: rpc: dial tcp 104.36.82.49:4005: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:55:57.170524104 (61h20m6.893087155s) [noise=true , address=104.36.82.15:4289 , piece_id=MFZHE2Q766VXNQ43QOIPXO33RASSYDVTJ2QLTVR4ACCHMGJLVV2A, seg_pos={0 0}, upload=58793]: Done uploading piece. err: ecclient: upload failed (node:126WPF6MtX7YjuKf73rGJrifdXyd78KY4FdL757qDipGNHBW6uY, address:104.36.82.15:4289): protocol: rpc: dial tcp 104.36.82.15:4289: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:55:57.170556958 (61h20m6.893119998s) [seg_pos={0 0}, upload=58793]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Jul 27 00:55:57.400256958 (61h20m7.122819998s) [upload=58793]: Done uploading segments... Jul 27 00:55:57.400281876 (61h20m7.122844916s) [upload=58793]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 5.90480419s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [.X..X.X..............X......................................] duration: 5.534127223s FAIL (2025-07-27 01:22:20 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.534153836s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 9 out of 60 is failed (threshold=3) [.XXXX.....X....X....XXX.....................................] duration: 5.936690236s FAIL (2025-07-27 01:22:19 UTC) orbiter warning: uplink: context deadline exceeded Duration: 5.936719844s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [XXXXXX..X.X.X..........X....................................] duration: 5.734573667s FAIL (2025-07-27 01:22:17 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== 7326QE72JPEXPXFHPPYHSNKIURIMRGIQ, seg_pos={0 0}, upload=58778]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR): protocol: write tcp 188.166.225.170:33562->104.36.82.11:4115: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:54:03.385284649 (61h18m13.107847679s) [seg_pos={0 0}, upload=58778]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 27 00:54:03.385367337 (61h18m13.107930377s) [upload=58778]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 27 00:54:03.385371374 (61h18m13.107934414s) [upload=58778]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 27 00:54:03.688084400 (61h18m13.41064744s) [upload=58778]: Done uploading segments... Jul 27 00:54:03.688101954 (61h18m13.410664994s) [upload=58778]: Upload finished. err: <nil> Jul 27 00:54:29.969393323 (61h18m39.691956364s) [upload=58786]: Starting upload Jul 27 00:54:29.969484158 (61h18m39.692047198s) [upload=58786]: Done starting upload Jul 27 00:54:29.969609477 (61h18m39.692172517s) [upload=58786]: Uploading segments... Jul 27 00:54:29.969791546 (61h18m39.692354576s) [upload=58786]: Got next segment. Inline: false Jul 27 00:54:29.969897859 (61h18m39.6924609s) [upload=58786]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 00:54:29.969903801 (61h18m39.692466841s) [upload=58786]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 00:54:30.202493651 (61h18m39.925056691s) [seg_pos={0 0}, upload=58786]: Begin upload segment... Jul 27 00:54:30.202544298 (61h18m39.925107338s) [seg_pos={0 0}, upload=58786]: Done begin upload segment. Jul 27 00:54:30.202565960 (61h18m39.92512899s) [noise=true , address=104.36.82.15:4259 , piece_id=2CYXGFMVV7Z4PMRHEJ23VWSK53BM3C4DRQPNSPOI55MY7FLS2OSQ, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.202629651 (61h18m39.925192691s) [noise=true , address=45.91.116.5:4299 , piece_id=5K6GYVCVQ46VZRUMNZ3PEITFO66VABEGLAV7NZCV5M36QVUVMFCA, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.202926890 (61h18m39.92548992s) [noise=true , address=45.91.116.7:4010 , piece_id=EKECSPS64QHNIO3EIRAVAT35KA5TCPBMDBLT5TSB3LZWKP6KZZHA, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.203102106 (61h18m39.925665146s) [noise=true , address=77.90.31.16:4735 , piece_id=N32CSF74OC4RGUPMJST5IV5P3FI6RBEUHKXCPXVNUFNHHSN5ZH4A, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.203204351 (61h18m39.925767391s) [noise=true , address=104.36.82.5:4838 , piece_id=JNEB4FUOK534MWNRW4DB5VJPPCUIBDA6FIX2YXWSRXNFQ6GDACHA, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.203445583 (61h18m39.926008623s) [noise=true , address=77.90.31.19:4843 , piece_id=KHZD4RVLBISBTIER5RAE2UCFCEFYYWCPMMXLSKIRUTIQTMLODQXQ, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.203491891 (61h18m39.926054921s) [noise=true , address=77.90.31.10:4520 , piece_id=4GRJDJRCNJPKJ5VV23VRZRNBLAOCJEHFVW534O3M3TQ4SRSP2FEQ, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.203718445 (61h18m39.926281475s) [noise=true , address=45.91.116.6:4974 , piece_id=LR3SRAZFX3NSX7KTZOR643UHIIJSXI4XTBCXYA6W2SFKSEBN7Q2Q, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.204010243 (61h18m39.926573283s) [noise=true , address=45.91.116.12:4227 , piece_id=J2OI4W4CT2FLAUB7WOGOAMTRWRS4PB5AQXLF4XFZNNO2J6LL46SA, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.204257396 (61h18m39.926820436s) [noise=true , address=77.90.31.12:4592 , piece_id=EA7S4T5VCRCJBRETDY6FJ43SOP7NWEVMR2F6L64J3TB6TNOOCKAA, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.204507925 (61h18m39.927070976s) [noise=true , address=45.91.116.4:4939 , piece_id=ZR6IIF5N5GSW6XIMVVRFVAE56M4HNLL5J7SSJE5M5UUFXKZFJSFQ, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.205281365 (61h18m39.927844435s) [noise=true , address=45.91.116.6:4973 , piece_id=VKQD7SENX3EDE34BXHGL4Z7MTET2OWSFLTOZ4PPYTDD2QVUHZEJQ, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.205688314 (61h18m39.928251354s) [noise=true , address=77.90.31.14:4662 , piece_id=VBVLHQKNOIU25REDYLILDDF3XRZFFU5W5IY4I7YMUE2WTZXLBHEQ, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.205981905 (61h18m39.928544945s) [noise=true , address=45.91.116.7:4009 , piece_id=4V2UC4QPMJKHBAI7BVHTVPZQFY667XVV3DCXAN3GKL444IHXEVEA, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.206218718 (61h18m39.928781759s) [noise=true , address=104.36.82.9:4005 , piece_id=3ARJN3YOZQ7Y3YYCSDQOHV5PEWB7WWJSS3WCHLJ52KWKD32CUKDQ, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.207082942 (61h18m39.929645982s) [noise=true , address=104.36.82.7:4910 , piece_id=ZJYKL4SR6BZP6OYD7QQOZADAR7GDVKVXV4GERU6FQE52RXZGCGOA, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:31.892857589 (61h18m41.615420639s) [upload=58786]: Next returned nil segment Jul 27 00:54:31.892877968 (61h18m41.615441008s) [upload=58786]: Waiting for error group managing segments... Jul 27 00:54:32.223482398 (61h18m41.946045438s) [noise=true , address=45.91.116.6:4974 , piece_id=LR3SRAZFX3NSX7KTZOR643UHIIJSXI4XTBCXYA6W2SFKSEBN7Q2Q, seg_pos={0 0}, upload=58786]: Done uploading piece. err: <nil> Jul 27 00:54:32.237060637 (61h18m41.959623677s) [noise=true , address=45.91.116.4:4939 , piece_id=ZR6IIF5N5GSW6XIMVVRFVAE56M4HNLL5J7SSJE5M5UUFXKZFJSFQ, seg_pos={0 0}, upload=58786]: Done uploading piece. err: <nil> Jul 27 00:54:32.299221539 (61h18m42.021784619s) [noise=true , address=45.91.116.7:4009 , piece_id=4V2UC4QPMJKHBAI7BVHTVPZQFY667XVV3DCXAN3GKL444IHXEVEA, seg_pos={0 0}, upload=58786]: Done uploading piece. err: <nil> Jul 27 00:54:32.325716521 (61h18m42.048279561s) [noise=true , address=45.91.116.6:4973 , piece_id=VKQD7SENX3EDE34BXHGL4Z7MTET2OWSFLTOZ4PPYTDD2QVUHZEJQ, seg_pos={0 0}, upload=58786]: Done uploading piece. err: <nil> Jul 27 00:54:32.325884473 (61h18m42.048447503s) [noise=true , address=45.91.116.12:4227 , piece_id=J2OI4W4CT2FLAUB7WOGOAMTRWRS4PB5AQXLF4XFZNNO2J6LL46SA, seg_pos={0 0}, upload=58786]: Done uploading piece. err: <nil> Jul 27 00:54:32.372983466 (61h18m42.095546506s) [noise=true , address=45.91.116.5:4299 , piece_id=5K6GYVCVQ46VZRUMNZ3PEITFO66VABEGLAV7NZCV5M36QVUVMFCA, seg_pos={0 0}, upload=58786]: Done uploading piece. err: <nil> Jul 27 00:54:32.379844103 (61h18m42.102407133s) [noise=true , address=45.91.116.7:4010 , piece_id=EKECSPS64QHNIO3EIRAVAT35KA5TCPBMDBLT5TSB3LZWKP6KZZHA, seg_pos={0 0}, upload=58786]: Done uploading piece. err: <nil> Jul 27 00:54:32.384666850 (61h18m42.10722989s) [noise=true , address=77.90.31.14:4662 , piece_id=VBVLHQKNOIU25REDYLILDDF3XRZFFU5W5IY4I7YMUE2WTZXLBHEQ, seg_pos={0 0}, upload=58786]: Done uploading piece. err: <nil> Jul 27 00:54:32.472371113 (61h18m42.194934143s) [noise=true , address=77.90.31.16:4735 , piece_id=N32CSF74OC4RGUPMJST5IV5P3FI6RBEUHKXCPXVNUFNHHSN5ZH4A, seg_pos={0 0}, upload=58786]: Done uploading piece. err: <nil> Jul 27 00:54:33.667049984 (61h18m43.389613024s) [noise=true , address=77.90.31.19:4843 , piece_id=KHZD4RVLBISBTIER5RAE2UCFCEFYYWCPMMXLSKIRUTIQTMLODQXQ, seg_pos={0 0}, upload=58786]: Done uploading piece. err: <nil> Jul 27 00:54:34.213712174 (61h18m43.936275214s) [noise=true , address=77.90.31.12:4592 , piece_id=EA7S4T5VCRCJBRETDY6FJ43SOP7NWEVMR2F6L64J3TB6TNOOCKAA, seg_pos={0 0}, upload=58786]: Done uploading piece. err: <nil> Jul 27 00:54:42.252927986 (61h18m51.975491026s) [noise=true , address=77.90.31.10:4520 , piece_id=4GRJDJRCNJPKJ5VV23VRZRNBLAOCJEHFVW534O3M3TQ4SRSP2FEQ, seg_pos={0 0}, upload=58786]: Done uploading piece. err: <nil> Jul 27 00:54:59.328639826 (61h19m9.051202866s) [noise=true , address=104.36.82.5:4838 , piece_id=JNEB4FUOK534MWNRW4DB5VJPPCUIBDA6FIX2YXWSRXNFQ6GDACHA, seg_pos={0 0}, upload=58786]: Done uploading piece. err: ecclient: upload failed (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL, address:104.36.82.5:4838): protocol: rpc: dial tcp 104.36.82.5:4838: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:54:59.328713146 (61h19m9.051276186s) [noise=true , address=104.36.82.7:4910 , piece_id=ZJYKL4SR6BZP6OYD7QQOZADAR7GDVKVXV4GERU6FQE52RXZGCGOA, seg_pos={0 0}, upload=58786]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:104.36.82.7:4910): protocol: rpc: dial tcp 104.36.82.7:4910: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:54:59.328897458 (61h19m9.051460488s) [noise=true , address=104.36.82.15:4259 , piece_id=2CYXGFMVV7Z4PMRHEJ23VWSK53BM3C4DRQPNSPOI55MY7FLS2OSQ, seg_pos={0 0}, upload=58786]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:104.36.82.15:4259): protocol: write tcp 188.166.225.170:42398->104.36.82.15:4259: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:54:59.329496955 (61h19m9.052059995s) [noise=true , address=104.36.82.9:4005 , piece_id=3ARJN3YOZQ7Y3YYCSDQOHV5PEWB7WWJSS3WCHLJ52KWKD32CUKDQ, seg_pos={0 0}, upload=58786]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:104.36.82.9:4005): protocol: rpc: dial tcp 104.36.82.9:4005: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:54:59.329528055 (61h19m9.052091095s) [seg_pos={0 0}, upload=58786]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 27 00:54:59.557689493 (61h19m9.280252533s) [upload=58786]: Done uploading segments... Jul 27 00:54:59.557710713 (61h19m9.280273753s) [upload=58786]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.734600347s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 4 out of 60 is failed (threshold=3) [...................X....X..........X...................X....] duration: 1.851812749s FAIL (2025-07-27 01:22:02 UTC) orbiter warning: digest mismatch: after 792f8e52f0f261131431c48786048ce73833f71a202ca313ecdd5b39aa603e1c != before 4c0418a6860f4eb95c9171181a1c1a28de1d7886deefc8fd7b2ed1a91cded834 Duration: 1.851845642s core uplink us1 vivint
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [..X..X.X..............X.....................................] duration: 5.635017659s FAIL (2025-07-27 01:21:49 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.635043798s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 4 out of 60 is failed (threshold=3) [....................X....X..........X...................X...] duration: 2.032757661s FAIL (2025-07-27 01:21:34 UTC) orbiter warning: digest mismatch: after 792f8e52f0f261131431c48786048ce73833f71a202ca313ecdd5b39aa603e1c != before 4c0418a6860f4eb95c9171181a1c1a28de1d7886deefc8fd7b2ed1a91cded834 Duration: 2.032781917s core uplink us1 vivint
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 5 out of 60 is failed (threshold=3) [X..X..X.X..............X....................................] duration: 5.785259927s FAIL (2025-07-27 01:21:31 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.78527679s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 11 out of 60 is failed (threshold=3) [XXXXXXX..X.X.X..........X...................................] duration: 5.767083891s FAIL (2025-07-27 01:21:28 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== 7326QE72JPEXPXFHPPYHSNKIURIMRGIQ, seg_pos={0 0}, upload=58778]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR): protocol: write tcp 188.166.225.170:33562->104.36.82.11:4115: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:54:03.385284649 (61h18m13.107847679s) [seg_pos={0 0}, upload=58778]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 27 00:54:03.385367337 (61h18m13.107930377s) [upload=58778]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 27 00:54:03.385371374 (61h18m13.107934414s) [upload=58778]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 27 00:54:03.688084400 (61h18m13.41064744s) [upload=58778]: Done uploading segments... Jul 27 00:54:03.688101954 (61h18m13.410664994s) [upload=58778]: Upload finished. err: <nil> Jul 27 00:54:29.969393323 (61h18m39.691956364s) [upload=58786]: Starting upload Jul 27 00:54:29.969484158 (61h18m39.692047198s) [upload=58786]: Done starting upload Jul 27 00:54:29.969609477 (61h18m39.692172517s) [upload=58786]: Uploading segments... Jul 27 00:54:29.969791546 (61h18m39.692354576s) [upload=58786]: Got next segment. Inline: false Jul 27 00:54:29.969897859 (61h18m39.6924609s) [upload=58786]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 00:54:29.969903801 (61h18m39.692466841s) [upload=58786]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 00:54:30.202493651 (61h18m39.925056691s) [seg_pos={0 0}, upload=58786]: Begin upload segment... Jul 27 00:54:30.202544298 (61h18m39.925107338s) [seg_pos={0 0}, upload=58786]: Done begin upload segment. Jul 27 00:54:30.202565960 (61h18m39.92512899s) [noise=true , address=104.36.82.15:4259 , piece_id=2CYXGFMVV7Z4PMRHEJ23VWSK53BM3C4DRQPNSPOI55MY7FLS2OSQ, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.202629651 (61h18m39.925192691s) [noise=true , address=45.91.116.5:4299 , piece_id=5K6GYVCVQ46VZRUMNZ3PEITFO66VABEGLAV7NZCV5M36QVUVMFCA, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.202926890 (61h18m39.92548992s) [noise=true , address=45.91.116.7:4010 , piece_id=EKECSPS64QHNIO3EIRAVAT35KA5TCPBMDBLT5TSB3LZWKP6KZZHA, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.203102106 (61h18m39.925665146s) [noise=true , address=77.90.31.16:4735 , piece_id=N32CSF74OC4RGUPMJST5IV5P3FI6RBEUHKXCPXVNUFNHHSN5ZH4A, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.203204351 (61h18m39.925767391s) [noise=true , address=104.36.82.5:4838 , piece_id=JNEB4FUOK534MWNRW4DB5VJPPCUIBDA6FIX2YXWSRXNFQ6GDACHA, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.203445583 (61h18m39.926008623s) [noise=true , address=77.90.31.19:4843 , piece_id=KHZD4RVLBISBTIER5RAE2UCFCEFYYWCPMMXLSKIRUTIQTMLODQXQ, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.203491891 (61h18m39.926054921s) [noise=true , address=77.90.31.10:4520 , piece_id=4GRJDJRCNJPKJ5VV23VRZRNBLAOCJEHFVW534O3M3TQ4SRSP2FEQ, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.203718445 (61h18m39.926281475s) [noise=true , address=45.91.116.6:4974 , piece_id=LR3SRAZFX3NSX7KTZOR643UHIIJSXI4XTBCXYA6W2SFKSEBN7Q2Q, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.204010243 (61h18m39.926573283s) [noise=true , address=45.91.116.12:4227 , piece_id=J2OI4W4CT2FLAUB7WOGOAMTRWRS4PB5AQXLF4XFZNNO2J6LL46SA, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.204257396 (61h18m39.926820436s) [noise=true , address=77.90.31.12:4592 , piece_id=EA7S4T5VCRCJBRETDY6FJ43SOP7NWEVMR2F6L64J3TB6TNOOCKAA, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.204507925 (61h18m39.927070976s) [noise=true , address=45.91.116.4:4939 , piece_id=ZR6IIF5N5GSW6XIMVVRFVAE56M4HNLL5J7SSJE5M5UUFXKZFJSFQ, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.205281365 (61h18m39.927844435s) [noise=true , address=45.91.116.6:4973 , piece_id=VKQD7SENX3EDE34BXHGL4Z7MTET2OWSFLTOZ4PPYTDD2QVUHZEJQ, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.205688314 (61h18m39.928251354s) [noise=true , address=77.90.31.14:4662 , piece_id=VBVLHQKNOIU25REDYLILDDF3XRZFFU5W5IY4I7YMUE2WTZXLBHEQ, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.205981905 (61h18m39.928544945s) [noise=true , address=45.91.116.7:4009 , piece_id=4V2UC4QPMJKHBAI7BVHTVPZQFY667XVV3DCXAN3GKL444IHXEVEA, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.206218718 (61h18m39.928781759s) [noise=true , address=104.36.82.9:4005 , piece_id=3ARJN3YOZQ7Y3YYCSDQOHV5PEWB7WWJSS3WCHLJ52KWKD32CUKDQ, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.207082942 (61h18m39.929645982s) [noise=true , address=104.36.82.7:4910 , piece_id=ZJYKL4SR6BZP6OYD7QQOZADAR7GDVKVXV4GERU6FQE52RXZGCGOA, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:31.892857589 (61h18m41.615420639s) [upload=58786]: Next returned nil segment Jul 27 00:54:31.892877968 (61h18m41.615441008s) [upload=58786]: Waiting for error group managing segments... Jul 27 00:54:32.223482398 (61h18m41.946045438s) [noise=true , address=45.91.116.6:4974 , piece_id=LR3SRAZFX3NSX7KTZOR643UHIIJSXI4XTBCXYA6W2SFKSEBN7Q2Q, seg_pos={0 0}, upload=58786]: Done uploading piece. err: <nil> Jul 27 00:54:32.237060637 (61h18m41.959623677s) [noise=true , address=45.91.116.4:4939 , piece_id=ZR6IIF5N5GSW6XIMVVRFVAE56M4HNLL5J7SSJE5M5UUFXKZFJSFQ, seg_pos={0 0}, upload=58786]: Done uploading piece. err: <nil> Jul 27 00:54:32.299221539 (61h18m42.021784619s) [noise=true , address=45.91.116.7:4009 , piece_id=4V2UC4QPMJKHBAI7BVHTVPZQFY667XVV3DCXAN3GKL444IHXEVEA, seg_pos={0 0}, upload=58786]: Done uploading piece. err: <nil> Jul 27 00:54:32.325716521 (61h18m42.048279561s) [noise=true , address=45.91.116.6:4973 , piece_id=VKQD7SENX3EDE34BXHGL4Z7MTET2OWSFLTOZ4PPYTDD2QVUHZEJQ, seg_pos={0 0}, upload=58786]: Done uploading piece. err: <nil> Jul 27 00:54:32.325884473 (61h18m42.048447503s) [noise=true , address=45.91.116.12:4227 , piece_id=J2OI4W4CT2FLAUB7WOGOAMTRWRS4PB5AQXLF4XFZNNO2J6LL46SA, seg_pos={0 0}, upload=58786]: Done uploading piece. err: <nil> Jul 27 00:54:32.372983466 (61h18m42.095546506s) [noise=true , address=45.91.116.5:4299 , piece_id=5K6GYVCVQ46VZRUMNZ3PEITFO66VABEGLAV7NZCV5M36QVUVMFCA, seg_pos={0 0}, upload=58786]: Done uploading piece. err: <nil> Jul 27 00:54:32.379844103 (61h18m42.102407133s) [noise=true , address=45.91.116.7:4010 , piece_id=EKECSPS64QHNIO3EIRAVAT35KA5TCPBMDBLT5TSB3LZWKP6KZZHA, seg_pos={0 0}, upload=58786]: Done uploading piece. err: <nil> Jul 27 00:54:32.384666850 (61h18m42.10722989s) [noise=true , address=77.90.31.14:4662 , piece_id=VBVLHQKNOIU25REDYLILDDF3XRZFFU5W5IY4I7YMUE2WTZXLBHEQ, seg_pos={0 0}, upload=58786]: Done uploading piece. err: <nil> Jul 27 00:54:32.472371113 (61h18m42.194934143s) [noise=true , address=77.90.31.16:4735 , piece_id=N32CSF74OC4RGUPMJST5IV5P3FI6RBEUHKXCPXVNUFNHHSN5ZH4A, seg_pos={0 0}, upload=58786]: Done uploading piece. err: <nil> Jul 27 00:54:33.667049984 (61h18m43.389613024s) [noise=true , address=77.90.31.19:4843 , piece_id=KHZD4RVLBISBTIER5RAE2UCFCEFYYWCPMMXLSKIRUTIQTMLODQXQ, seg_pos={0 0}, upload=58786]: Done uploading piece. err: <nil> Jul 27 00:54:34.213712174 (61h18m43.936275214s) [noise=true , address=77.90.31.12:4592 , piece_id=EA7S4T5VCRCJBRETDY6FJ43SOP7NWEVMR2F6L64J3TB6TNOOCKAA, seg_pos={0 0}, upload=58786]: Done uploading piece. err: <nil> Jul 27 00:54:42.252927986 (61h18m51.975491026s) [noise=true , address=77.90.31.10:4520 , piece_id=4GRJDJRCNJPKJ5VV23VRZRNBLAOCJEHFVW534O3M3TQ4SRSP2FEQ, seg_pos={0 0}, upload=58786]: Done uploading piece. err: <nil> Jul 27 00:54:59.328639826 (61h19m9.051202866s) [noise=true , address=104.36.82.5:4838 , piece_id=JNEB4FUOK534MWNRW4DB5VJPPCUIBDA6FIX2YXWSRXNFQ6GDACHA, seg_pos={0 0}, upload=58786]: Done uploading piece. err: ecclient: upload failed (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL, address:104.36.82.5:4838): protocol: rpc: dial tcp 104.36.82.5:4838: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:54:59.328713146 (61h19m9.051276186s) [noise=true , address=104.36.82.7:4910 , piece_id=ZJYKL4SR6BZP6OYD7QQOZADAR7GDVKVXV4GERU6FQE52RXZGCGOA, seg_pos={0 0}, upload=58786]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:104.36.82.7:4910): protocol: rpc: dial tcp 104.36.82.7:4910: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:54:59.328897458 (61h19m9.051460488s) [noise=true , address=104.36.82.15:4259 , piece_id=2CYXGFMVV7Z4PMRHEJ23VWSK53BM3C4DRQPNSPOI55MY7FLS2OSQ, seg_pos={0 0}, upload=58786]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:104.36.82.15:4259): protocol: write tcp 188.166.225.170:42398->104.36.82.15:4259: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:54:59.329496955 (61h19m9.052059995s) [noise=true , address=104.36.82.9:4005 , piece_id=3ARJN3YOZQ7Y3YYCSDQOHV5PEWB7WWJSS3WCHLJ52KWKD32CUKDQ, seg_pos={0 0}, upload=58786]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:104.36.82.9:4005): protocol: rpc: dial tcp 104.36.82.9:4005: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:54:59.329528055 (61h19m9.052091095s) [seg_pos={0 0}, upload=58786]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 27 00:54:59.557689493 (61h19m9.280252533s) [upload=58786]: Done uploading segments... Jul 27 00:54:59.557710713 (61h19m9.280273753s) [upload=58786]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.767110281s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 15 out of 60 is failed (threshold=3) [XXXXXXXXX.......XXXXXX......................................] duration: 5.818918835s FAIL (2025-07-27 01:21:27 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== false Jul 27 00:54:57.821101936 (61h19m7.543664977s) [upload=58793]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 00:54:57.821108168 (61h19m7.543671208s) [upload=58793]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 00:54:58.268999381 (61h19m7.991562411s) [seg_pos={0 0}, upload=58793]: Begin upload segment... Jul 27 00:54:58.269047784 (61h19m7.991610824s) [seg_pos={0 0}, upload=58793]: Done begin upload segment. Jul 27 00:54:58.269068714 (61h19m7.991631754s) [noise=true , address=79.127.201.215:4344 , piece_id=BFA6WIJYTUHSVLKQVLXWCBKNYJJB6CXUUVTZMZEESHH4QG3OV4NQ, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.269108290 (61h19m7.99167133s) [noise=true , address=79.127.205.235:4682 , piece_id=4BXTF4IATE3AD46376DXSOM25E3UL75T4DYKSI7YKKAY4XDUGXZQ, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.269416039 (61h19m7.991979079s) [noise=true , address=104.36.82.53:4153 , piece_id=S2YS2I44RPQEKILNXGKZ6JKUWZIU47ZJTQTDDH6IKCKCEL7IM2NA, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.269588800 (61h19m7.99215184s) [noise=true , address=64.49.13.137:4502 , piece_id=COTTEJEOI7INJS3ZDBHS2CLIGMREMHZGHN7MB22TJR2Y4LZMVRRA, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.269666007 (61h19m7.992229048s) [noise=true , address=77.90.31.11:4573 , piece_id=352UN4WWJYSEOQ4VR4CDGD5XWCDY6TXNXZJW6COACYDDPNTGZSOA, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.269938439 (61h19m7.992501479s) [noise=true , address=104.36.82.46:4500 , piece_id=SQPQJFK3QRENUZNGARKNCCVDHDY2ZXIZNVOFZV76AC63YFACQTBQ, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.270186764 (61h19m7.992749794s) [noise=true , address=79.127.219.38:4001 , piece_id=ZKBCPE6IA6NQJDRGDGFQT2NNGIXAH64ZZ27OE2WPUR54XOQQ2Q3Q, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.270456489 (61h19m7.993019529s) [noise=true , address=104.36.82.47:4545 , piece_id=F4LGWDTZ3CMZIZBDGZZ4TGZ6GKFYE3GNW2B6SZEXBBMYYFS6Y3CQ, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.270693122 (61h19m7.993256162s) [noise=true , address=104.36.82.15:4289 , piece_id=MFZHE2Q766VXNQ43QOIPXO33RASSYDVTJ2QLTVR4ACCHMGJLVV2A, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.270924895 (61h19m7.993487935s) [noise=true , address=77.90.31.10:4525 , piece_id=B7APUOSESFZRV2QUHZ6KZLR7VRJTPHNZ3FFWMEPYV6O7I7EQFVDA, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.270973609 (61h19m7.993536649s) [noise=true , address=104.36.82.52:4091 , piece_id=6Z4ZKBKTEMXCSMNIC6277XZMSJPBKTFW5RJ2PC726VVJTALRJ6NA, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.271180806 (61h19m7.993743846s) [noise=true , address=79.127.205.245:4242 , piece_id=MH52WQ77OW4ZDQFVVFICHDWERB3WGGZXONOOA66J4P32ILZAREQQ, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.271367583 (61h19m7.993930613s) [noise=true , address=64.49.13.140:4514 , piece_id=RXOR5I3R3HF6WUJPKES4OQFTPMETGOEFPZB2KTUS4WVSQUIQQTDA, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.271609946 (61h19m7.994172986s) [noise=true , address=156.146.43.225:4448 , piece_id=TAPXLBQNT5ERYAWMWFBHQF4LYXAZJS4XXIAHXT57S3ORLKEX4MQQ, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.271884882 (61h19m7.994447922s) [noise=true , address=104.36.82.49:4005 , piece_id=J6YISXBRJWWSOJLDNE4U72EQFENIS7SPYZM4SCEVJG2SUGDHRNDA, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.271925130 (61h19m7.99448817s) [noise=true , address=79.127.219.47:4072 , piece_id=JBTVVQE3YIPDNKFAWIPDB5UBQZXP7KJJW4OBCXTAZDUQWLODTZWQ, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.272230464 (61h19m7.994793504s) [noise=true , address=77.90.31.16:4751 , piece_id=XC46QZVCMKTVTLQYDPGM4ZRRD4KDIFCJTQMHFDMY633FQG74FWWQ, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.272352857 (61h19m7.994915898s) [noise=true , address=79.127.220.101:4041 , piece_id=DBBBRJX5NPIPW5P72KSEANTJRZEM5MMUJQQTLXDSCYVUQY6IIIXA, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:59.403085511 (61h19m9.125648551s) [upload=58793]: Next returned nil segment Jul 27 00:54:59.403106221 (61h19m9.125669281s) [upload=58793]: Waiting for error group managing segments... Jul 27 00:54:59.699317478 (61h19m9.421880518s) [noise=true , address=64.49.13.137:4502 , piece_id=COTTEJEOI7INJS3ZDBHS2CLIGMREMHZGHN7MB22TJR2Y4LZMVRRA, seg_pos={0 0}, upload=58793]: Done uploading piece. err: <nil> Jul 27 00:54:59.712155049 (61h19m9.434718079s) [noise=true , address=79.127.205.235:4682 , piece_id=4BXTF4IATE3AD46376DXSOM25E3UL75T4DYKSI7YKKAY4XDUGXZQ, seg_pos={0 0}, upload=58793]: Done uploading piece. err: <nil> Jul 27 00:54:59.718874015 (61h19m9.441437055s) [noise=true , address=79.127.205.245:4242 , piece_id=MH52WQ77OW4ZDQFVVFICHDWERB3WGGZXONOOA66J4P32ILZAREQQ, seg_pos={0 0}, upload=58793]: Done uploading piece. err: <nil> Jul 27 00:55:00.219903397 (61h19m9.942466437s) [noise=true , address=77.90.31.16:4751 , piece_id=XC46QZVCMKTVTLQYDPGM4ZRRD4KDIFCJTQMHFDMY633FQG74FWWQ, seg_pos={0 0}, upload=58793]: Done uploading piece. err: <nil> Jul 27 00:55:00.393745135 (61h19m10.116308165s) [noise=true , address=77.90.31.11:4573 , piece_id=352UN4WWJYSEOQ4VR4CDGD5XWCDY6TXNXZJW6COACYDDPNTGZSOA, seg_pos={0 0}, upload=58793]: Done uploading piece. err: <nil> Jul 27 00:55:01.589079921 (61h19m11.311642972s) [noise=true , address=79.127.219.47:4072 , piece_id=JBTVVQE3YIPDNKFAWIPDB5UBQZXP7KJJW4OBCXTAZDUQWLODTZWQ, seg_pos={0 0}, upload=58793]: Done uploading piece. err: <nil> Jul 27 00:55:01.684825540 (61h19m11.40738858s) [noise=true , address=79.127.220.101:4041 , piece_id=DBBBRJX5NPIPW5P72KSEANTJRZEM5MMUJQQTLXDSCYVUQY6IIIXA, seg_pos={0 0}, upload=58793]: Done uploading piece. err: <nil> Jul 27 00:55:02.296389006 (61h19m12.018952046s) [noise=true , address=156.146.43.225:4448 , piece_id=TAPXLBQNT5ERYAWMWFBHQF4LYXAZJS4XXIAHXT57S3ORLKEX4MQQ, seg_pos={0 0}, upload=58793]: Done uploading piece. err: <nil> Jul 27 00:55:03.015602295 (61h19m12.738165335s) [noise=true , address=79.127.219.38:4001 , piece_id=ZKBCPE6IA6NQJDRGDGFQT2NNGIXAH64ZZ27OE2WPUR54XOQQ2Q3Q, seg_pos={0 0}, upload=58793]: Done uploading piece. err: <nil> Jul 27 00:55:03.588380603 (61h19m13.310943643s) [noise=true , address=77.90.31.10:4525 , piece_id=B7APUOSESFZRV2QUHZ6KZLR7VRJTPHNZ3FFWMEPYV6O7I7EQFVDA, seg_pos={0 0}, upload=58793]: Done uploading piece. err: <nil> Jul 27 00:55:07.419707501 (61h19m17.142270581s) [noise=true , address=79.127.201.215:4344 , piece_id=BFA6WIJYTUHSVLKQVLXWCBKNYJJB6CXUUVTZMZEESHH4QG3OV4NQ, seg_pos={0 0}, upload=58793]: Done uploading piece. err: <nil> Jul 27 00:55:10.064731586 (61h19m19.787294626s) [noise=true , address=64.49.13.140:4514 , piece_id=RXOR5I3R3HF6WUJPKES4OQFTPMETGOEFPZB2KTUS4WVSQUIQQTDA, seg_pos={0 0}, upload=58793]: Done uploading piece. err: <nil> Jul 27 00:55:57.170253087 (61h20m6.892816127s) [noise=true , address=104.36.82.53:4153 , piece_id=S2YS2I44RPQEKILNXGKZ6JKUWZIU47ZJTQTDDH6IKCKCEL7IM2NA, seg_pos={0 0}, upload=58793]: Done uploading piece. err: ecclient: upload failed (node:1fPp4DvLe18dZWPmMPFiC8o9jL4uiEFNmPAHHmFEH74ZXNndw3, address:104.36.82.53:4153): protocol: write tcp 188.166.225.170:41520->104.36.82.53:4153: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:55:57.170329522 (61h20m6.892892562s) [noise=true , address=104.36.82.47:4545 , piece_id=F4LGWDTZ3CMZIZBDGZZ4TGZ6GKFYE3GNW2B6SZEXBBMYYFS6Y3CQ, seg_pos={0 0}, upload=58793]: Done uploading piece. err: ecclient: upload failed (node:12euvmhGegnPYufoKKQ1on9FTv2AhuicSVYuWyXtDiY9BLdDMYE, address:104.36.82.47:4545): protocol: rpc: dial tcp 104.36.82.47:4545: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:55:57.170370181 (61h20m6.892933221s) [noise=true , address=104.36.82.52:4091 , piece_id=6Z4ZKBKTEMXCSMNIC6277XZMSJPBKTFW5RJ2PC726VVJTALRJ6NA, seg_pos={0 0}, upload=58793]: Done uploading piece. err: ecclient: upload failed (node:17wSQzrFc2hUimSsNwDxCgDDi4TpHkbCEH38fGKf9B8WqWzxpW, address:104.36.82.52:4091): protocol: rpc: dial tcp 104.36.82.52:4091: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:55:57.170401270 (61h20m6.89296431s) [noise=true , address=104.36.82.46:4500 , piece_id=SQPQJFK3QRENUZNGARKNCCVDHDY2ZXIZNVOFZV76AC63YFACQTBQ, seg_pos={0 0}, upload=58793]: Done uploading piece. err: ecclient: upload failed (node:12B7ox4ehqxJ2mwqeXj8C9N9GuSU5etNC23iT1pJnTgMLGBAGQK, address:104.36.82.46:4500): protocol: rpc: dial tcp 104.36.82.46:4500: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:55:57.170514156 (61h20m6.893077196s) [noise=true , address=104.36.82.49:4005 , piece_id=J6YISXBRJWWSOJLDNE4U72EQFENIS7SPYZM4SCEVJG2SUGDHRNDA, seg_pos={0 0}, upload=58793]: Done uploading piece. err: ecclient: upload failed (node:1AivThSVLAp9zneNK79sic4AqnTQ62yvNTFF36imigarMGqDSW, address:104.36.82.49:4005): protocol: rpc: dial tcp 104.36.82.49:4005: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:55:57.170524104 (61h20m6.893087155s) [noise=true , address=104.36.82.15:4289 , piece_id=MFZHE2Q766VXNQ43QOIPXO33RASSYDVTJ2QLTVR4ACCHMGJLVV2A, seg_pos={0 0}, upload=58793]: Done uploading piece. err: ecclient: upload failed (node:126WPF6MtX7YjuKf73rGJrifdXyd78KY4FdL757qDipGNHBW6uY, address:104.36.82.15:4289): protocol: rpc: dial tcp 104.36.82.15:4289: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:55:57.170556958 (61h20m6.893119998s) [seg_pos={0 0}, upload=58793]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Jul 27 00:55:57.400256958 (61h20m7.122819998s) [upload=58793]: Done uploading segments... Jul 27 00:55:57.400281876 (61h20m7.122844916s) [upload=58793]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 5.818944113s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 9 out of 60 is failed (threshold=3) [..XXXX.....X....X....XXX....................................] duration: 5.715824617s FAIL (2025-07-27 01:21:10 UTC) orbiter warning: uplink: context deadline exceeded Duration: 5.715850828s core uplink us1 vivint
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 6 out of 60 is failed (threshold=3) [XX..X..X.X..............X...................................] duration: 5.627888421s FAIL (2025-07-27 01:21:05 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.627911364s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 16 out of 60 is failed (threshold=3) [XXXXXXXXXX.......XXXXXX.....................................] duration: 5.791439872s FAIL (2025-07-27 01:21:00 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== false Jul 27 00:54:57.821101936 (61h19m7.543664977s) [upload=58793]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 00:54:57.821108168 (61h19m7.543671208s) [upload=58793]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 00:54:58.268999381 (61h19m7.991562411s) [seg_pos={0 0}, upload=58793]: Begin upload segment... Jul 27 00:54:58.269047784 (61h19m7.991610824s) [seg_pos={0 0}, upload=58793]: Done begin upload segment. Jul 27 00:54:58.269068714 (61h19m7.991631754s) [noise=true , address=79.127.201.215:4344 , piece_id=BFA6WIJYTUHSVLKQVLXWCBKNYJJB6CXUUVTZMZEESHH4QG3OV4NQ, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.269108290 (61h19m7.99167133s) [noise=true , address=79.127.205.235:4682 , piece_id=4BXTF4IATE3AD46376DXSOM25E3UL75T4DYKSI7YKKAY4XDUGXZQ, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.269416039 (61h19m7.991979079s) [noise=true , address=104.36.82.53:4153 , piece_id=S2YS2I44RPQEKILNXGKZ6JKUWZIU47ZJTQTDDH6IKCKCEL7IM2NA, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.269588800 (61h19m7.99215184s) [noise=true , address=64.49.13.137:4502 , piece_id=COTTEJEOI7INJS3ZDBHS2CLIGMREMHZGHN7MB22TJR2Y4LZMVRRA, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.269666007 (61h19m7.992229048s) [noise=true , address=77.90.31.11:4573 , piece_id=352UN4WWJYSEOQ4VR4CDGD5XWCDY6TXNXZJW6COACYDDPNTGZSOA, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.269938439 (61h19m7.992501479s) [noise=true , address=104.36.82.46:4500 , piece_id=SQPQJFK3QRENUZNGARKNCCVDHDY2ZXIZNVOFZV76AC63YFACQTBQ, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.270186764 (61h19m7.992749794s) [noise=true , address=79.127.219.38:4001 , piece_id=ZKBCPE6IA6NQJDRGDGFQT2NNGIXAH64ZZ27OE2WPUR54XOQQ2Q3Q, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.270456489 (61h19m7.993019529s) [noise=true , address=104.36.82.47:4545 , piece_id=F4LGWDTZ3CMZIZBDGZZ4TGZ6GKFYE3GNW2B6SZEXBBMYYFS6Y3CQ, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.270693122 (61h19m7.993256162s) [noise=true , address=104.36.82.15:4289 , piece_id=MFZHE2Q766VXNQ43QOIPXO33RASSYDVTJ2QLTVR4ACCHMGJLVV2A, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.270924895 (61h19m7.993487935s) [noise=true , address=77.90.31.10:4525 , piece_id=B7APUOSESFZRV2QUHZ6KZLR7VRJTPHNZ3FFWMEPYV6O7I7EQFVDA, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.270973609 (61h19m7.993536649s) [noise=true , address=104.36.82.52:4091 , piece_id=6Z4ZKBKTEMXCSMNIC6277XZMSJPBKTFW5RJ2PC726VVJTALRJ6NA, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.271180806 (61h19m7.993743846s) [noise=true , address=79.127.205.245:4242 , piece_id=MH52WQ77OW4ZDQFVVFICHDWERB3WGGZXONOOA66J4P32ILZAREQQ, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.271367583 (61h19m7.993930613s) [noise=true , address=64.49.13.140:4514 , piece_id=RXOR5I3R3HF6WUJPKES4OQFTPMETGOEFPZB2KTUS4WVSQUIQQTDA, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.271609946 (61h19m7.994172986s) [noise=true , address=156.146.43.225:4448 , piece_id=TAPXLBQNT5ERYAWMWFBHQF4LYXAZJS4XXIAHXT57S3ORLKEX4MQQ, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.271884882 (61h19m7.994447922s) [noise=true , address=104.36.82.49:4005 , piece_id=J6YISXBRJWWSOJLDNE4U72EQFENIS7SPYZM4SCEVJG2SUGDHRNDA, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.271925130 (61h19m7.99448817s) [noise=true , address=79.127.219.47:4072 , piece_id=JBTVVQE3YIPDNKFAWIPDB5UBQZXP7KJJW4OBCXTAZDUQWLODTZWQ, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.272230464 (61h19m7.994793504s) [noise=true , address=77.90.31.16:4751 , piece_id=XC46QZVCMKTVTLQYDPGM4ZRRD4KDIFCJTQMHFDMY633FQG74FWWQ, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.272352857 (61h19m7.994915898s) [noise=true , address=79.127.220.101:4041 , piece_id=DBBBRJX5NPIPW5P72KSEANTJRZEM5MMUJQQTLXDSCYVUQY6IIIXA, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:59.403085511 (61h19m9.125648551s) [upload=58793]: Next returned nil segment Jul 27 00:54:59.403106221 (61h19m9.125669281s) [upload=58793]: Waiting for error group managing segments... Jul 27 00:54:59.699317478 (61h19m9.421880518s) [noise=true , address=64.49.13.137:4502 , piece_id=COTTEJEOI7INJS3ZDBHS2CLIGMREMHZGHN7MB22TJR2Y4LZMVRRA, seg_pos={0 0}, upload=58793]: Done uploading piece. err: <nil> Jul 27 00:54:59.712155049 (61h19m9.434718079s) [noise=true , address=79.127.205.235:4682 , piece_id=4BXTF4IATE3AD46376DXSOM25E3UL75T4DYKSI7YKKAY4XDUGXZQ, seg_pos={0 0}, upload=58793]: Done uploading piece. err: <nil> Jul 27 00:54:59.718874015 (61h19m9.441437055s) [noise=true , address=79.127.205.245:4242 , piece_id=MH52WQ77OW4ZDQFVVFICHDWERB3WGGZXONOOA66J4P32ILZAREQQ, seg_pos={0 0}, upload=58793]: Done uploading piece. err: <nil> Jul 27 00:55:00.219903397 (61h19m9.942466437s) [noise=true , address=77.90.31.16:4751 , piece_id=XC46QZVCMKTVTLQYDPGM4ZRRD4KDIFCJTQMHFDMY633FQG74FWWQ, seg_pos={0 0}, upload=58793]: Done uploading piece. err: <nil> Jul 27 00:55:00.393745135 (61h19m10.116308165s) [noise=true , address=77.90.31.11:4573 , piece_id=352UN4WWJYSEOQ4VR4CDGD5XWCDY6TXNXZJW6COACYDDPNTGZSOA, seg_pos={0 0}, upload=58793]: Done uploading piece. err: <nil> Jul 27 00:55:01.589079921 (61h19m11.311642972s) [noise=true , address=79.127.219.47:4072 , piece_id=JBTVVQE3YIPDNKFAWIPDB5UBQZXP7KJJW4OBCXTAZDUQWLODTZWQ, seg_pos={0 0}, upload=58793]: Done uploading piece. err: <nil> Jul 27 00:55:01.684825540 (61h19m11.40738858s) [noise=true , address=79.127.220.101:4041 , piece_id=DBBBRJX5NPIPW5P72KSEANTJRZEM5MMUJQQTLXDSCYVUQY6IIIXA, seg_pos={0 0}, upload=58793]: Done uploading piece. err: <nil> Jul 27 00:55:02.296389006 (61h19m12.018952046s) [noise=true , address=156.146.43.225:4448 , piece_id=TAPXLBQNT5ERYAWMWFBHQF4LYXAZJS4XXIAHXT57S3ORLKEX4MQQ, seg_pos={0 0}, upload=58793]: Done uploading piece. err: <nil> Jul 27 00:55:03.015602295 (61h19m12.738165335s) [noise=true , address=79.127.219.38:4001 , piece_id=ZKBCPE6IA6NQJDRGDGFQT2NNGIXAH64ZZ27OE2WPUR54XOQQ2Q3Q, seg_pos={0 0}, upload=58793]: Done uploading piece. err: <nil> Jul 27 00:55:03.588380603 (61h19m13.310943643s) [noise=true , address=77.90.31.10:4525 , piece_id=B7APUOSESFZRV2QUHZ6KZLR7VRJTPHNZ3FFWMEPYV6O7I7EQFVDA, seg_pos={0 0}, upload=58793]: Done uploading piece. err: <nil> Jul 27 00:55:07.419707501 (61h19m17.142270581s) [noise=true , address=79.127.201.215:4344 , piece_id=BFA6WIJYTUHSVLKQVLXWCBKNYJJB6CXUUVTZMZEESHH4QG3OV4NQ, seg_pos={0 0}, upload=58793]: Done uploading piece. err: <nil> Jul 27 00:55:10.064731586 (61h19m19.787294626s) [noise=true , address=64.49.13.140:4514 , piece_id=RXOR5I3R3HF6WUJPKES4OQFTPMETGOEFPZB2KTUS4WVSQUIQQTDA, seg_pos={0 0}, upload=58793]: Done uploading piece. err: <nil> Jul 27 00:55:57.170253087 (61h20m6.892816127s) [noise=true , address=104.36.82.53:4153 , piece_id=S2YS2I44RPQEKILNXGKZ6JKUWZIU47ZJTQTDDH6IKCKCEL7IM2NA, seg_pos={0 0}, upload=58793]: Done uploading piece. err: ecclient: upload failed (node:1fPp4DvLe18dZWPmMPFiC8o9jL4uiEFNmPAHHmFEH74ZXNndw3, address:104.36.82.53:4153): protocol: write tcp 188.166.225.170:41520->104.36.82.53:4153: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:55:57.170329522 (61h20m6.892892562s) [noise=true , address=104.36.82.47:4545 , piece_id=F4LGWDTZ3CMZIZBDGZZ4TGZ6GKFYE3GNW2B6SZEXBBMYYFS6Y3CQ, seg_pos={0 0}, upload=58793]: Done uploading piece. err: ecclient: upload failed (node:12euvmhGegnPYufoKKQ1on9FTv2AhuicSVYuWyXtDiY9BLdDMYE, address:104.36.82.47:4545): protocol: rpc: dial tcp 104.36.82.47:4545: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:55:57.170370181 (61h20m6.892933221s) [noise=true , address=104.36.82.52:4091 , piece_id=6Z4ZKBKTEMXCSMNIC6277XZMSJPBKTFW5RJ2PC726VVJTALRJ6NA, seg_pos={0 0}, upload=58793]: Done uploading piece. err: ecclient: upload failed (node:17wSQzrFc2hUimSsNwDxCgDDi4TpHkbCEH38fGKf9B8WqWzxpW, address:104.36.82.52:4091): protocol: rpc: dial tcp 104.36.82.52:4091: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:55:57.170401270 (61h20m6.89296431s) [noise=true , address=104.36.82.46:4500 , piece_id=SQPQJFK3QRENUZNGARKNCCVDHDY2ZXIZNVOFZV76AC63YFACQTBQ, seg_pos={0 0}, upload=58793]: Done uploading piece. err: ecclient: upload failed (node:12B7ox4ehqxJ2mwqeXj8C9N9GuSU5etNC23iT1pJnTgMLGBAGQK, address:104.36.82.46:4500): protocol: rpc: dial tcp 104.36.82.46:4500: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:55:57.170514156 (61h20m6.893077196s) [noise=true , address=104.36.82.49:4005 , piece_id=J6YISXBRJWWSOJLDNE4U72EQFENIS7SPYZM4SCEVJG2SUGDHRNDA, seg_pos={0 0}, upload=58793]: Done uploading piece. err: ecclient: upload failed (node:1AivThSVLAp9zneNK79sic4AqnTQ62yvNTFF36imigarMGqDSW, address:104.36.82.49:4005): protocol: rpc: dial tcp 104.36.82.49:4005: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:55:57.170524104 (61h20m6.893087155s) [noise=true , address=104.36.82.15:4289 , piece_id=MFZHE2Q766VXNQ43QOIPXO33RASSYDVTJ2QLTVR4ACCHMGJLVV2A, seg_pos={0 0}, upload=58793]: Done uploading piece. err: ecclient: upload failed (node:126WPF6MtX7YjuKf73rGJrifdXyd78KY4FdL757qDipGNHBW6uY, address:104.36.82.15:4289): protocol: rpc: dial tcp 104.36.82.15:4289: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:55:57.170556958 (61h20m6.893119998s) [seg_pos={0 0}, upload=58793]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Jul 27 00:55:57.400256958 (61h20m7.122819998s) [upload=58793]: Done uploading segments... Jul 27 00:55:57.400281876 (61h20m7.122844916s) [upload=58793]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 5.791465851s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 4 out of 60 is failed (threshold=3) [.....................X....X..........X...................X..] duration: 2.05220482s FAIL (2025-07-27 01:20:45 UTC) orbiter warning: digest mismatch: after 792f8e52f0f261131431c48786048ce73833f71a202ca313ecdd5b39aa603e1c != before 4c0418a6860f4eb95c9171181a1c1a28de1d7886deefc8fd7b2ed1a91cded834 Duration: 2.052234015s core uplink us1 vivint

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)