Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................X.........................................] duration: 21.76071739s FAIL (2025-09-07 11:07:08 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Sep 7 10:37:37.259810434 (21h9m13.299383554s) [upload=18635]: Starting upload Sep 7 10:37:37.259869848 (21h9m13.299442968s) [upload=18635]: Done starting upload Sep 7 10:37:37.259991862 (21h9m13.299564982s) [upload=18635]: Uploading segments... Sep 7 10:37:37.260091512 (21h9m13.299664632s) [upload=18635]: Got next segment. Inline: false Sep 7 10:37:37.260124284 (21h9m13.299697405s) [upload=18635]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 7 10:37:37.260129676 (21h9m13.299702796s) [upload=18635]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 7 10:37:37.523765418 (21h9m13.563338538s) [seg_pos={0 0}, upload=18635]: Begin upload segment... Sep 7 10:37:37.523848977 (21h9m13.563422098s) [seg_pos={0 0}, upload=18635]: Done begin upload segment. Sep 7 10:37:37.523875308 (21h9m13.563448418s) [noise=true , address=45.91.116.11:4119 , piece_id=Y5TR5QSARCJ3HXQZN6OWO7NTLMSGCSLCOEZVJRSB3VZQQL7O57NA, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.523927959 (21h9m13.563501079s) [noise=true , address=45.91.116.5:4298 , piece_id=T4KK43URSFHH2LDPBX4CN66G4KVBCWV7JSJM3CADPG6EYGOS2HFQ, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.524262138 (21h9m13.563835258s) [noise=true , address=45.91.116.6:4973 , piece_id=X63L2GPVONW7ORT5DYOQIHV46EG4DIHEB6TLOZMXKLCQPR5EWDFA, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.524469726 (21h9m13.564042846s) [noise=true , address=45.91.116.6:4974 , piece_id=ZUB4DD6ZXILUSYESNMGNOCKZVTPUY24N52YF2KY3Z36JYNDGRM2A, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.524553666 (21h9m13.564126786s) [noise=true , address=45.91.116.4:4939 , piece_id=BXLXHF3W5CPAS73RZXC4GL4PPN5PRMVLSSW4ADSW2M2KWRXQY77Q, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.524760001 (21h9m13.564333121s) [noise=true , address=77.90.31.10:4519 , piece_id=5RGPSBJO5NDUDD7Y4NH5CYTGF2FB5RO6ADKEP7YUGZFVMCTACVKA, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.524839723 (21h9m13.564412843s) [noise=true , address=45.91.116.12:4225 , piece_id=JQOKRSTWFE56HRHV4BBEL3ZR7T24XBHVYYU4WA4SWJGF432EPOLQ, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.525132163 (21h9m13.564705293s) [noise=true , address=45.91.116.12:4227 , piece_id=7BZ46KAIEIJDIKHXRLDCMB4Y6E7KSC4QLBOLY2A2NZFA4GVVXD2A, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.525296006 (21h9m13.564869127s) [noise=true , address=45.91.116.6:4975 , piece_id=7U2D53SCSJ5GUGLFWEH6FA66Q5U7EIK7YVBTP5GP5P5OOAVOHQWA, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.525386890 (21h9m13.56496001s) [noise=true , address=45.91.116.10:4083 , piece_id=2XGLF7IS7LC7TU5X2M27UEYQBVX3UML3UEWSFUISJOXF6AVKITTQ, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.525611531 (21h9m13.565184651s) [noise=true , address=45.91.116.7:4009 , piece_id=TYKXIJX7XIMFYKEA5RMORUFVO7UDVKNVBSJAIFUEIB3A76G4HN3Q, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.525648361 (21h9m13.565221481s) [noise=true , address=45.91.116.14:4191 , piece_id=BLZ4EZD5DQ7QQNNRLNCKAB2CQBZGQ2SYKGJJIOZCXNGS4SYOFNXA, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.525870506 (21h9m13.565443626s) [noise=true , address=45.91.116.11:4117 , piece_id=5E2JPHN5T2SETJNOGGAESLRU7PRPUCUDEO25QDW6PA73EKVGMRYQ, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.525935922 (21h9m13.565509042s) [noise=true , address=45.91.116.7:4010 , piece_id=UQRAOTS6AILKBAR6QC3ZD3VATJEYM2T6T4K2KDVU3XRKYQSEB66A, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.526222790 (21h9m13.56579591s) [noise=true , address=45.91.116.13:4153 , piece_id=KUSBCYGRA2XFXZPOYJG4EYGZ7FOSKPKV2EEFQHUND6TPTNQX7AGQ, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.526424746 (21h9m13.565997866s) [noise=true , address=77.90.31.10:4518 , piece_id=XBJTZKIXGIW3WPKU4RWYRZBMAC67YYDYRNAJNBWRYX6SHU3T73WA, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:39.362042598 (21h9m15.401615708s) [upload=18635]: Next returned nil segment Sep 7 10:37:39.362059750 (21h9m15.40163287s) [upload=18635]: Waiting for error group managing segments... Sep 7 10:37:39.669685089 (21h9m15.709258209s) [noise=true , address=45.91.116.14:4191 , piece_id=BLZ4EZD5DQ7QQNNRLNCKAB2CQBZGQ2SYKGJJIOZCXNGS4SYOFNXA, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.679437981 (21h9m15.719011101s) [noise=true , address=77.90.31.10:4519 , piece_id=5RGPSBJO5NDUDD7Y4NH5CYTGF2FB5RO6ADKEP7YUGZFVMCTACVKA, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.679555526 (21h9m15.719128646s) [noise=true , address=45.91.116.12:4225 , piece_id=JQOKRSTWFE56HRHV4BBEL3ZR7T24XBHVYYU4WA4SWJGF432EPOLQ, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.687220604 (21h9m15.726793724s) [noise=true , address=45.91.116.7:4010 , piece_id=UQRAOTS6AILKBAR6QC3ZD3VATJEYM2T6T4K2KDVU3XRKYQSEB66A, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.711092863 (21h9m15.750665983s) [noise=true , address=45.91.116.4:4939 , piece_id=BXLXHF3W5CPAS73RZXC4GL4PPN5PRMVLSSW4ADSW2M2KWRXQY77Q, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.734404882 (21h9m15.773978002s) [noise=true , address=45.91.116.7:4009 , piece_id=TYKXIJX7XIMFYKEA5RMORUFVO7UDVKNVBSJAIFUEIB3A76G4HN3Q, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.743143673 (21h9m15.782716793s) [noise=true , address=45.91.116.11:4119 , piece_id=Y5TR5QSARCJ3HXQZN6OWO7NTLMSGCSLCOEZVJRSB3VZQQL7O57NA, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.748574906 (21h9m15.788148016s) [noise=true , address=45.91.116.6:4974 , piece_id=ZUB4DD6ZXILUSYESNMGNOCKZVTPUY24N52YF2KY3Z36JYNDGRM2A, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.754311293 (21h9m15.793884413s) [noise=true , address=45.91.116.13:4153 , piece_id=KUSBCYGRA2XFXZPOYJG4EYGZ7FOSKPKV2EEFQHUND6TPTNQX7AGQ, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.755384276 (21h9m15.794957396s) [noise=true , address=45.91.116.5:4298 , piece_id=T4KK43URSFHH2LDPBX4CN66G4KVBCWV7JSJM3CADPG6EYGOS2HFQ, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.758010662 (21h9m15.797583782s) [noise=true , address=77.90.31.10:4518 , piece_id=XBJTZKIXGIW3WPKU4RWYRZBMAC67YYDYRNAJNBWRYX6SHU3T73WA, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.774473184 (21h9m15.814046304s) [noise=true , address=45.91.116.6:4973 , piece_id=X63L2GPVONW7ORT5DYOQIHV46EG4DIHEB6TLOZMXKLCQPR5EWDFA, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.817437797 (21h9m15.857010917s) [noise=true , address=45.91.116.11:4117 , piece_id=5E2JPHN5T2SETJNOGGAESLRU7PRPUCUDEO25QDW6PA73EKVGMRYQ, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.817462975 (21h9m15.857036096s) [seg_pos={0 0}, upload=18635]: Segment reached optimal threshold of 13 pieces. Sep 7 10:37:39.817597944 (21h9m15.857171064s) [noise=true , address=45.91.116.6:4975 , piece_id=7U2D53SCSJ5GUGLFWEH6FA66Q5U7EIK7YVBTP5GP5P5OOAVOHQWA, seg_pos={0 0}, upload=18635]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: expected piece hash; context canceled; EOF Sep 7 10:37:39.817683578 (21h9m15.857256698s) [noise=true , address=45.91.116.10:4083 , piece_id=2XGLF7IS7LC7TU5X2M27UEYQBVX3UML3UEWSFUISJOXF6AVKITTQ, seg_pos={0 0}, upload=18635]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: expected piece hash; context canceled; EOF Sep 7 10:37:39.817775434 (21h9m15.857348554s) [noise=true , address=45.91.116.12:4227 , piece_id=7BZ46KAIEIJDIKHXRLDCMB4Y6E7KSC4QLBOLY2A2NZFA4GVVXD2A, seg_pos={0 0}, upload=18635]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH): protocol: expected piece hash; context canceled; EOF Sep 7 10:37:39.817852270 (21h9m15.8574254s) [seg_pos={0 0}, upload=18635]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 7 10:37:39.817955087 (21h9m15.857528217s) [upload=18635]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 7 10:37:39.817961028 (21h9m15.857534148s) [upload=18635]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 7 10:37:40.298833319 (21h9m16.338406429s) [upload=18635]: Done uploading segments... Sep 7 10:37:40.298849160 (21h9m16.33842228s) [upload=18635]: Upload finished. err: <nil> Duration: 21.760782236s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................X........................................] duration: 6.481918072s FAIL (2025-09-07 11:06:05 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Sep 7 10:37:37.259810434 (21h9m13.299383554s) [upload=18635]: Starting upload Sep 7 10:37:37.259869848 (21h9m13.299442968s) [upload=18635]: Done starting upload Sep 7 10:37:37.259991862 (21h9m13.299564982s) [upload=18635]: Uploading segments... Sep 7 10:37:37.260091512 (21h9m13.299664632s) [upload=18635]: Got next segment. Inline: false Sep 7 10:37:37.260124284 (21h9m13.299697405s) [upload=18635]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 7 10:37:37.260129676 (21h9m13.299702796s) [upload=18635]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 7 10:37:37.523765418 (21h9m13.563338538s) [seg_pos={0 0}, upload=18635]: Begin upload segment... Sep 7 10:37:37.523848977 (21h9m13.563422098s) [seg_pos={0 0}, upload=18635]: Done begin upload segment. Sep 7 10:37:37.523875308 (21h9m13.563448418s) [noise=true , address=45.91.116.11:4119 , piece_id=Y5TR5QSARCJ3HXQZN6OWO7NTLMSGCSLCOEZVJRSB3VZQQL7O57NA, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.523927959 (21h9m13.563501079s) [noise=true , address=45.91.116.5:4298 , piece_id=T4KK43URSFHH2LDPBX4CN66G4KVBCWV7JSJM3CADPG6EYGOS2HFQ, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.524262138 (21h9m13.563835258s) [noise=true , address=45.91.116.6:4973 , piece_id=X63L2GPVONW7ORT5DYOQIHV46EG4DIHEB6TLOZMXKLCQPR5EWDFA, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.524469726 (21h9m13.564042846s) [noise=true , address=45.91.116.6:4974 , piece_id=ZUB4DD6ZXILUSYESNMGNOCKZVTPUY24N52YF2KY3Z36JYNDGRM2A, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.524553666 (21h9m13.564126786s) [noise=true , address=45.91.116.4:4939 , piece_id=BXLXHF3W5CPAS73RZXC4GL4PPN5PRMVLSSW4ADSW2M2KWRXQY77Q, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.524760001 (21h9m13.564333121s) [noise=true , address=77.90.31.10:4519 , piece_id=5RGPSBJO5NDUDD7Y4NH5CYTGF2FB5RO6ADKEP7YUGZFVMCTACVKA, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.524839723 (21h9m13.564412843s) [noise=true , address=45.91.116.12:4225 , piece_id=JQOKRSTWFE56HRHV4BBEL3ZR7T24XBHVYYU4WA4SWJGF432EPOLQ, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.525132163 (21h9m13.564705293s) [noise=true , address=45.91.116.12:4227 , piece_id=7BZ46KAIEIJDIKHXRLDCMB4Y6E7KSC4QLBOLY2A2NZFA4GVVXD2A, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.525296006 (21h9m13.564869127s) [noise=true , address=45.91.116.6:4975 , piece_id=7U2D53SCSJ5GUGLFWEH6FA66Q5U7EIK7YVBTP5GP5P5OOAVOHQWA, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.525386890 (21h9m13.56496001s) [noise=true , address=45.91.116.10:4083 , piece_id=2XGLF7IS7LC7TU5X2M27UEYQBVX3UML3UEWSFUISJOXF6AVKITTQ, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.525611531 (21h9m13.565184651s) [noise=true , address=45.91.116.7:4009 , piece_id=TYKXIJX7XIMFYKEA5RMORUFVO7UDVKNVBSJAIFUEIB3A76G4HN3Q, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.525648361 (21h9m13.565221481s) [noise=true , address=45.91.116.14:4191 , piece_id=BLZ4EZD5DQ7QQNNRLNCKAB2CQBZGQ2SYKGJJIOZCXNGS4SYOFNXA, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.525870506 (21h9m13.565443626s) [noise=true , address=45.91.116.11:4117 , piece_id=5E2JPHN5T2SETJNOGGAESLRU7PRPUCUDEO25QDW6PA73EKVGMRYQ, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.525935922 (21h9m13.565509042s) [noise=true , address=45.91.116.7:4010 , piece_id=UQRAOTS6AILKBAR6QC3ZD3VATJEYM2T6T4K2KDVU3XRKYQSEB66A, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.526222790 (21h9m13.56579591s) [noise=true , address=45.91.116.13:4153 , piece_id=KUSBCYGRA2XFXZPOYJG4EYGZ7FOSKPKV2EEFQHUND6TPTNQX7AGQ, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.526424746 (21h9m13.565997866s) [noise=true , address=77.90.31.10:4518 , piece_id=XBJTZKIXGIW3WPKU4RWYRZBMAC67YYDYRNAJNBWRYX6SHU3T73WA, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:39.362042598 (21h9m15.401615708s) [upload=18635]: Next returned nil segment Sep 7 10:37:39.362059750 (21h9m15.40163287s) [upload=18635]: Waiting for error group managing segments... Sep 7 10:37:39.669685089 (21h9m15.709258209s) [noise=true , address=45.91.116.14:4191 , piece_id=BLZ4EZD5DQ7QQNNRLNCKAB2CQBZGQ2SYKGJJIOZCXNGS4SYOFNXA, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.679437981 (21h9m15.719011101s) [noise=true , address=77.90.31.10:4519 , piece_id=5RGPSBJO5NDUDD7Y4NH5CYTGF2FB5RO6ADKEP7YUGZFVMCTACVKA, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.679555526 (21h9m15.719128646s) [noise=true , address=45.91.116.12:4225 , piece_id=JQOKRSTWFE56HRHV4BBEL3ZR7T24XBHVYYU4WA4SWJGF432EPOLQ, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.687220604 (21h9m15.726793724s) [noise=true , address=45.91.116.7:4010 , piece_id=UQRAOTS6AILKBAR6QC3ZD3VATJEYM2T6T4K2KDVU3XRKYQSEB66A, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.711092863 (21h9m15.750665983s) [noise=true , address=45.91.116.4:4939 , piece_id=BXLXHF3W5CPAS73RZXC4GL4PPN5PRMVLSSW4ADSW2M2KWRXQY77Q, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.734404882 (21h9m15.773978002s) [noise=true , address=45.91.116.7:4009 , piece_id=TYKXIJX7XIMFYKEA5RMORUFVO7UDVKNVBSJAIFUEIB3A76G4HN3Q, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.743143673 (21h9m15.782716793s) [noise=true , address=45.91.116.11:4119 , piece_id=Y5TR5QSARCJ3HXQZN6OWO7NTLMSGCSLCOEZVJRSB3VZQQL7O57NA, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.748574906 (21h9m15.788148016s) [noise=true , address=45.91.116.6:4974 , piece_id=ZUB4DD6ZXILUSYESNMGNOCKZVTPUY24N52YF2KY3Z36JYNDGRM2A, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.754311293 (21h9m15.793884413s) [noise=true , address=45.91.116.13:4153 , piece_id=KUSBCYGRA2XFXZPOYJG4EYGZ7FOSKPKV2EEFQHUND6TPTNQX7AGQ, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.755384276 (21h9m15.794957396s) [noise=true , address=45.91.116.5:4298 , piece_id=T4KK43URSFHH2LDPBX4CN66G4KVBCWV7JSJM3CADPG6EYGOS2HFQ, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.758010662 (21h9m15.797583782s) [noise=true , address=77.90.31.10:4518 , piece_id=XBJTZKIXGIW3WPKU4RWYRZBMAC67YYDYRNAJNBWRYX6SHU3T73WA, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.774473184 (21h9m15.814046304s) [noise=true , address=45.91.116.6:4973 , piece_id=X63L2GPVONW7ORT5DYOQIHV46EG4DIHEB6TLOZMXKLCQPR5EWDFA, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.817437797 (21h9m15.857010917s) [noise=true , address=45.91.116.11:4117 , piece_id=5E2JPHN5T2SETJNOGGAESLRU7PRPUCUDEO25QDW6PA73EKVGMRYQ, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.817462975 (21h9m15.857036096s) [seg_pos={0 0}, upload=18635]: Segment reached optimal threshold of 13 pieces. Sep 7 10:37:39.817597944 (21h9m15.857171064s) [noise=true , address=45.91.116.6:4975 , piece_id=7U2D53SCSJ5GUGLFWEH6FA66Q5U7EIK7YVBTP5GP5P5OOAVOHQWA, seg_pos={0 0}, upload=18635]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: expected piece hash; context canceled; EOF Sep 7 10:37:39.817683578 (21h9m15.857256698s) [noise=true , address=45.91.116.10:4083 , piece_id=2XGLF7IS7LC7TU5X2M27UEYQBVX3UML3UEWSFUISJOXF6AVKITTQ, seg_pos={0 0}, upload=18635]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: expected piece hash; context canceled; EOF Sep 7 10:37:39.817775434 (21h9m15.857348554s) [noise=true , address=45.91.116.12:4227 , piece_id=7BZ46KAIEIJDIKHXRLDCMB4Y6E7KSC4QLBOLY2A2NZFA4GVVXD2A, seg_pos={0 0}, upload=18635]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH): protocol: expected piece hash; context canceled; EOF Sep 7 10:37:39.817852270 (21h9m15.8574254s) [seg_pos={0 0}, upload=18635]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 7 10:37:39.817955087 (21h9m15.857528217s) [upload=18635]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 7 10:37:39.817961028 (21h9m15.857534148s) [upload=18635]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 7 10:37:40.298833319 (21h9m16.338406429s) [upload=18635]: Done uploading segments... Sep 7 10:37:40.298849160 (21h9m16.33842228s) [upload=18635]: Upload finished. err: <nil> Duration: 6.481945616s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................X.......................................] duration: 6.479533302s FAIL (2025-09-07 11:05:50 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Sep 7 10:37:37.259810434 (21h9m13.299383554s) [upload=18635]: Starting upload Sep 7 10:37:37.259869848 (21h9m13.299442968s) [upload=18635]: Done starting upload Sep 7 10:37:37.259991862 (21h9m13.299564982s) [upload=18635]: Uploading segments... Sep 7 10:37:37.260091512 (21h9m13.299664632s) [upload=18635]: Got next segment. Inline: false Sep 7 10:37:37.260124284 (21h9m13.299697405s) [upload=18635]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 7 10:37:37.260129676 (21h9m13.299702796s) [upload=18635]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 7 10:37:37.523765418 (21h9m13.563338538s) [seg_pos={0 0}, upload=18635]: Begin upload segment... Sep 7 10:37:37.523848977 (21h9m13.563422098s) [seg_pos={0 0}, upload=18635]: Done begin upload segment. Sep 7 10:37:37.523875308 (21h9m13.563448418s) [noise=true , address=45.91.116.11:4119 , piece_id=Y5TR5QSARCJ3HXQZN6OWO7NTLMSGCSLCOEZVJRSB3VZQQL7O57NA, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.523927959 (21h9m13.563501079s) [noise=true , address=45.91.116.5:4298 , piece_id=T4KK43URSFHH2LDPBX4CN66G4KVBCWV7JSJM3CADPG6EYGOS2HFQ, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.524262138 (21h9m13.563835258s) [noise=true , address=45.91.116.6:4973 , piece_id=X63L2GPVONW7ORT5DYOQIHV46EG4DIHEB6TLOZMXKLCQPR5EWDFA, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.524469726 (21h9m13.564042846s) [noise=true , address=45.91.116.6:4974 , piece_id=ZUB4DD6ZXILUSYESNMGNOCKZVTPUY24N52YF2KY3Z36JYNDGRM2A, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.524553666 (21h9m13.564126786s) [noise=true , address=45.91.116.4:4939 , piece_id=BXLXHF3W5CPAS73RZXC4GL4PPN5PRMVLSSW4ADSW2M2KWRXQY77Q, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.524760001 (21h9m13.564333121s) [noise=true , address=77.90.31.10:4519 , piece_id=5RGPSBJO5NDUDD7Y4NH5CYTGF2FB5RO6ADKEP7YUGZFVMCTACVKA, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.524839723 (21h9m13.564412843s) [noise=true , address=45.91.116.12:4225 , piece_id=JQOKRSTWFE56HRHV4BBEL3ZR7T24XBHVYYU4WA4SWJGF432EPOLQ, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.525132163 (21h9m13.564705293s) [noise=true , address=45.91.116.12:4227 , piece_id=7BZ46KAIEIJDIKHXRLDCMB4Y6E7KSC4QLBOLY2A2NZFA4GVVXD2A, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.525296006 (21h9m13.564869127s) [noise=true , address=45.91.116.6:4975 , piece_id=7U2D53SCSJ5GUGLFWEH6FA66Q5U7EIK7YVBTP5GP5P5OOAVOHQWA, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.525386890 (21h9m13.56496001s) [noise=true , address=45.91.116.10:4083 , piece_id=2XGLF7IS7LC7TU5X2M27UEYQBVX3UML3UEWSFUISJOXF6AVKITTQ, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.525611531 (21h9m13.565184651s) [noise=true , address=45.91.116.7:4009 , piece_id=TYKXIJX7XIMFYKEA5RMORUFVO7UDVKNVBSJAIFUEIB3A76G4HN3Q, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.525648361 (21h9m13.565221481s) [noise=true , address=45.91.116.14:4191 , piece_id=BLZ4EZD5DQ7QQNNRLNCKAB2CQBZGQ2SYKGJJIOZCXNGS4SYOFNXA, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.525870506 (21h9m13.565443626s) [noise=true , address=45.91.116.11:4117 , piece_id=5E2JPHN5T2SETJNOGGAESLRU7PRPUCUDEO25QDW6PA73EKVGMRYQ, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.525935922 (21h9m13.565509042s) [noise=true , address=45.91.116.7:4010 , piece_id=UQRAOTS6AILKBAR6QC3ZD3VATJEYM2T6T4K2KDVU3XRKYQSEB66A, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.526222790 (21h9m13.56579591s) [noise=true , address=45.91.116.13:4153 , piece_id=KUSBCYGRA2XFXZPOYJG4EYGZ7FOSKPKV2EEFQHUND6TPTNQX7AGQ, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.526424746 (21h9m13.565997866s) [noise=true , address=77.90.31.10:4518 , piece_id=XBJTZKIXGIW3WPKU4RWYRZBMAC67YYDYRNAJNBWRYX6SHU3T73WA, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:39.362042598 (21h9m15.401615708s) [upload=18635]: Next returned nil segment Sep 7 10:37:39.362059750 (21h9m15.40163287s) [upload=18635]: Waiting for error group managing segments... Sep 7 10:37:39.669685089 (21h9m15.709258209s) [noise=true , address=45.91.116.14:4191 , piece_id=BLZ4EZD5DQ7QQNNRLNCKAB2CQBZGQ2SYKGJJIOZCXNGS4SYOFNXA, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.679437981 (21h9m15.719011101s) [noise=true , address=77.90.31.10:4519 , piece_id=5RGPSBJO5NDUDD7Y4NH5CYTGF2FB5RO6ADKEP7YUGZFVMCTACVKA, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.679555526 (21h9m15.719128646s) [noise=true , address=45.91.116.12:4225 , piece_id=JQOKRSTWFE56HRHV4BBEL3ZR7T24XBHVYYU4WA4SWJGF432EPOLQ, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.687220604 (21h9m15.726793724s) [noise=true , address=45.91.116.7:4010 , piece_id=UQRAOTS6AILKBAR6QC3ZD3VATJEYM2T6T4K2KDVU3XRKYQSEB66A, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.711092863 (21h9m15.750665983s) [noise=true , address=45.91.116.4:4939 , piece_id=BXLXHF3W5CPAS73RZXC4GL4PPN5PRMVLSSW4ADSW2M2KWRXQY77Q, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.734404882 (21h9m15.773978002s) [noise=true , address=45.91.116.7:4009 , piece_id=TYKXIJX7XIMFYKEA5RMORUFVO7UDVKNVBSJAIFUEIB3A76G4HN3Q, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.743143673 (21h9m15.782716793s) [noise=true , address=45.91.116.11:4119 , piece_id=Y5TR5QSARCJ3HXQZN6OWO7NTLMSGCSLCOEZVJRSB3VZQQL7O57NA, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.748574906 (21h9m15.788148016s) [noise=true , address=45.91.116.6:4974 , piece_id=ZUB4DD6ZXILUSYESNMGNOCKZVTPUY24N52YF2KY3Z36JYNDGRM2A, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.754311293 (21h9m15.793884413s) [noise=true , address=45.91.116.13:4153 , piece_id=KUSBCYGRA2XFXZPOYJG4EYGZ7FOSKPKV2EEFQHUND6TPTNQX7AGQ, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.755384276 (21h9m15.794957396s) [noise=true , address=45.91.116.5:4298 , piece_id=T4KK43URSFHH2LDPBX4CN66G4KVBCWV7JSJM3CADPG6EYGOS2HFQ, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.758010662 (21h9m15.797583782s) [noise=true , address=77.90.31.10:4518 , piece_id=XBJTZKIXGIW3WPKU4RWYRZBMAC67YYDYRNAJNBWRYX6SHU3T73WA, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.774473184 (21h9m15.814046304s) [noise=true , address=45.91.116.6:4973 , piece_id=X63L2GPVONW7ORT5DYOQIHV46EG4DIHEB6TLOZMXKLCQPR5EWDFA, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.817437797 (21h9m15.857010917s) [noise=true , address=45.91.116.11:4117 , piece_id=5E2JPHN5T2SETJNOGGAESLRU7PRPUCUDEO25QDW6PA73EKVGMRYQ, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.817462975 (21h9m15.857036096s) [seg_pos={0 0}, upload=18635]: Segment reached optimal threshold of 13 pieces. Sep 7 10:37:39.817597944 (21h9m15.857171064s) [noise=true , address=45.91.116.6:4975 , piece_id=7U2D53SCSJ5GUGLFWEH6FA66Q5U7EIK7YVBTP5GP5P5OOAVOHQWA, seg_pos={0 0}, upload=18635]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: expected piece hash; context canceled; EOF Sep 7 10:37:39.817683578 (21h9m15.857256698s) [noise=true , address=45.91.116.10:4083 , piece_id=2XGLF7IS7LC7TU5X2M27UEYQBVX3UML3UEWSFUISJOXF6AVKITTQ, seg_pos={0 0}, upload=18635]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: expected piece hash; context canceled; EOF Sep 7 10:37:39.817775434 (21h9m15.857348554s) [noise=true , address=45.91.116.12:4227 , piece_id=7BZ46KAIEIJDIKHXRLDCMB4Y6E7KSC4QLBOLY2A2NZFA4GVVXD2A, seg_pos={0 0}, upload=18635]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH): protocol: expected piece hash; context canceled; EOF Sep 7 10:37:39.817852270 (21h9m15.8574254s) [seg_pos={0 0}, upload=18635]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 7 10:37:39.817955087 (21h9m15.857528217s) [upload=18635]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 7 10:37:39.817961028 (21h9m15.857534148s) [upload=18635]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 7 10:37:40.298833319 (21h9m16.338406429s) [upload=18635]: Done uploading segments... Sep 7 10:37:40.298849160 (21h9m16.33842228s) [upload=18635]: Upload finished. err: <nil> Duration: 6.479563119s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................X......................................] duration: 8.38964461s FAIL (2025-09-07 11:05:34 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Sep 7 10:37:37.259810434 (21h9m13.299383554s) [upload=18635]: Starting upload Sep 7 10:37:37.259869848 (21h9m13.299442968s) [upload=18635]: Done starting upload Sep 7 10:37:37.259991862 (21h9m13.299564982s) [upload=18635]: Uploading segments... Sep 7 10:37:37.260091512 (21h9m13.299664632s) [upload=18635]: Got next segment. Inline: false Sep 7 10:37:37.260124284 (21h9m13.299697405s) [upload=18635]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 7 10:37:37.260129676 (21h9m13.299702796s) [upload=18635]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 7 10:37:37.523765418 (21h9m13.563338538s) [seg_pos={0 0}, upload=18635]: Begin upload segment... Sep 7 10:37:37.523848977 (21h9m13.563422098s) [seg_pos={0 0}, upload=18635]: Done begin upload segment. Sep 7 10:37:37.523875308 (21h9m13.563448418s) [noise=true , address=45.91.116.11:4119 , piece_id=Y5TR5QSARCJ3HXQZN6OWO7NTLMSGCSLCOEZVJRSB3VZQQL7O57NA, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.523927959 (21h9m13.563501079s) [noise=true , address=45.91.116.5:4298 , piece_id=T4KK43URSFHH2LDPBX4CN66G4KVBCWV7JSJM3CADPG6EYGOS2HFQ, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.524262138 (21h9m13.563835258s) [noise=true , address=45.91.116.6:4973 , piece_id=X63L2GPVONW7ORT5DYOQIHV46EG4DIHEB6TLOZMXKLCQPR5EWDFA, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.524469726 (21h9m13.564042846s) [noise=true , address=45.91.116.6:4974 , piece_id=ZUB4DD6ZXILUSYESNMGNOCKZVTPUY24N52YF2KY3Z36JYNDGRM2A, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.524553666 (21h9m13.564126786s) [noise=true , address=45.91.116.4:4939 , piece_id=BXLXHF3W5CPAS73RZXC4GL4PPN5PRMVLSSW4ADSW2M2KWRXQY77Q, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.524760001 (21h9m13.564333121s) [noise=true , address=77.90.31.10:4519 , piece_id=5RGPSBJO5NDUDD7Y4NH5CYTGF2FB5RO6ADKEP7YUGZFVMCTACVKA, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.524839723 (21h9m13.564412843s) [noise=true , address=45.91.116.12:4225 , piece_id=JQOKRSTWFE56HRHV4BBEL3ZR7T24XBHVYYU4WA4SWJGF432EPOLQ, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.525132163 (21h9m13.564705293s) [noise=true , address=45.91.116.12:4227 , piece_id=7BZ46KAIEIJDIKHXRLDCMB4Y6E7KSC4QLBOLY2A2NZFA4GVVXD2A, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.525296006 (21h9m13.564869127s) [noise=true , address=45.91.116.6:4975 , piece_id=7U2D53SCSJ5GUGLFWEH6FA66Q5U7EIK7YVBTP5GP5P5OOAVOHQWA, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.525386890 (21h9m13.56496001s) [noise=true , address=45.91.116.10:4083 , piece_id=2XGLF7IS7LC7TU5X2M27UEYQBVX3UML3UEWSFUISJOXF6AVKITTQ, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.525611531 (21h9m13.565184651s) [noise=true , address=45.91.116.7:4009 , piece_id=TYKXIJX7XIMFYKEA5RMORUFVO7UDVKNVBSJAIFUEIB3A76G4HN3Q, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.525648361 (21h9m13.565221481s) [noise=true , address=45.91.116.14:4191 , piece_id=BLZ4EZD5DQ7QQNNRLNCKAB2CQBZGQ2SYKGJJIOZCXNGS4SYOFNXA, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.525870506 (21h9m13.565443626s) [noise=true , address=45.91.116.11:4117 , piece_id=5E2JPHN5T2SETJNOGGAESLRU7PRPUCUDEO25QDW6PA73EKVGMRYQ, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.525935922 (21h9m13.565509042s) [noise=true , address=45.91.116.7:4010 , piece_id=UQRAOTS6AILKBAR6QC3ZD3VATJEYM2T6T4K2KDVU3XRKYQSEB66A, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.526222790 (21h9m13.56579591s) [noise=true , address=45.91.116.13:4153 , piece_id=KUSBCYGRA2XFXZPOYJG4EYGZ7FOSKPKV2EEFQHUND6TPTNQX7AGQ, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.526424746 (21h9m13.565997866s) [noise=true , address=77.90.31.10:4518 , piece_id=XBJTZKIXGIW3WPKU4RWYRZBMAC67YYDYRNAJNBWRYX6SHU3T73WA, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:39.362042598 (21h9m15.401615708s) [upload=18635]: Next returned nil segment Sep 7 10:37:39.362059750 (21h9m15.40163287s) [upload=18635]: Waiting for error group managing segments... Sep 7 10:37:39.669685089 (21h9m15.709258209s) [noise=true , address=45.91.116.14:4191 , piece_id=BLZ4EZD5DQ7QQNNRLNCKAB2CQBZGQ2SYKGJJIOZCXNGS4SYOFNXA, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.679437981 (21h9m15.719011101s) [noise=true , address=77.90.31.10:4519 , piece_id=5RGPSBJO5NDUDD7Y4NH5CYTGF2FB5RO6ADKEP7YUGZFVMCTACVKA, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.679555526 (21h9m15.719128646s) [noise=true , address=45.91.116.12:4225 , piece_id=JQOKRSTWFE56HRHV4BBEL3ZR7T24XBHVYYU4WA4SWJGF432EPOLQ, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.687220604 (21h9m15.726793724s) [noise=true , address=45.91.116.7:4010 , piece_id=UQRAOTS6AILKBAR6QC3ZD3VATJEYM2T6T4K2KDVU3XRKYQSEB66A, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.711092863 (21h9m15.750665983s) [noise=true , address=45.91.116.4:4939 , piece_id=BXLXHF3W5CPAS73RZXC4GL4PPN5PRMVLSSW4ADSW2M2KWRXQY77Q, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.734404882 (21h9m15.773978002s) [noise=true , address=45.91.116.7:4009 , piece_id=TYKXIJX7XIMFYKEA5RMORUFVO7UDVKNVBSJAIFUEIB3A76G4HN3Q, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.743143673 (21h9m15.782716793s) [noise=true , address=45.91.116.11:4119 , piece_id=Y5TR5QSARCJ3HXQZN6OWO7NTLMSGCSLCOEZVJRSB3VZQQL7O57NA, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.748574906 (21h9m15.788148016s) [noise=true , address=45.91.116.6:4974 , piece_id=ZUB4DD6ZXILUSYESNMGNOCKZVTPUY24N52YF2KY3Z36JYNDGRM2A, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.754311293 (21h9m15.793884413s) [noise=true , address=45.91.116.13:4153 , piece_id=KUSBCYGRA2XFXZPOYJG4EYGZ7FOSKPKV2EEFQHUND6TPTNQX7AGQ, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.755384276 (21h9m15.794957396s) [noise=true , address=45.91.116.5:4298 , piece_id=T4KK43URSFHH2LDPBX4CN66G4KVBCWV7JSJM3CADPG6EYGOS2HFQ, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.758010662 (21h9m15.797583782s) [noise=true , address=77.90.31.10:4518 , piece_id=XBJTZKIXGIW3WPKU4RWYRZBMAC67YYDYRNAJNBWRYX6SHU3T73WA, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.774473184 (21h9m15.814046304s) [noise=true , address=45.91.116.6:4973 , piece_id=X63L2GPVONW7ORT5DYOQIHV46EG4DIHEB6TLOZMXKLCQPR5EWDFA, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.817437797 (21h9m15.857010917s) [noise=true , address=45.91.116.11:4117 , piece_id=5E2JPHN5T2SETJNOGGAESLRU7PRPUCUDEO25QDW6PA73EKVGMRYQ, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.817462975 (21h9m15.857036096s) [seg_pos={0 0}, upload=18635]: Segment reached optimal threshold of 13 pieces. Sep 7 10:37:39.817597944 (21h9m15.857171064s) [noise=true , address=45.91.116.6:4975 , piece_id=7U2D53SCSJ5GUGLFWEH6FA66Q5U7EIK7YVBTP5GP5P5OOAVOHQWA, seg_pos={0 0}, upload=18635]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: expected piece hash; context canceled; EOF Sep 7 10:37:39.817683578 (21h9m15.857256698s) [noise=true , address=45.91.116.10:4083 , piece_id=2XGLF7IS7LC7TU5X2M27UEYQBVX3UML3UEWSFUISJOXF6AVKITTQ, seg_pos={0 0}, upload=18635]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: expected piece hash; context canceled; EOF Sep 7 10:37:39.817775434 (21h9m15.857348554s) [noise=true , address=45.91.116.12:4227 , piece_id=7BZ46KAIEIJDIKHXRLDCMB4Y6E7KSC4QLBOLY2A2NZFA4GVVXD2A, seg_pos={0 0}, upload=18635]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH): protocol: expected piece hash; context canceled; EOF Sep 7 10:37:39.817852270 (21h9m15.8574254s) [seg_pos={0 0}, upload=18635]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 7 10:37:39.817955087 (21h9m15.857528217s) [upload=18635]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 7 10:37:39.817961028 (21h9m15.857534148s) [upload=18635]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 7 10:37:40.298833319 (21h9m16.338406429s) [upload=18635]: Done uploading segments... Sep 7 10:37:40.298849160 (21h9m16.33842228s) [upload=18635]: Upload finished. err: <nil> Duration: 8.389678666s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................X.....................................] duration: 6.862639775s FAIL (2025-09-07 11:04:39 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Sep 7 10:37:37.259810434 (21h9m13.299383554s) [upload=18635]: Starting upload Sep 7 10:37:37.259869848 (21h9m13.299442968s) [upload=18635]: Done starting upload Sep 7 10:37:37.259991862 (21h9m13.299564982s) [upload=18635]: Uploading segments... Sep 7 10:37:37.260091512 (21h9m13.299664632s) [upload=18635]: Got next segment. Inline: false Sep 7 10:37:37.260124284 (21h9m13.299697405s) [upload=18635]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 7 10:37:37.260129676 (21h9m13.299702796s) [upload=18635]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 7 10:37:37.523765418 (21h9m13.563338538s) [seg_pos={0 0}, upload=18635]: Begin upload segment... Sep 7 10:37:37.523848977 (21h9m13.563422098s) [seg_pos={0 0}, upload=18635]: Done begin upload segment. Sep 7 10:37:37.523875308 (21h9m13.563448418s) [noise=true , address=45.91.116.11:4119 , piece_id=Y5TR5QSARCJ3HXQZN6OWO7NTLMSGCSLCOEZVJRSB3VZQQL7O57NA, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.523927959 (21h9m13.563501079s) [noise=true , address=45.91.116.5:4298 , piece_id=T4KK43URSFHH2LDPBX4CN66G4KVBCWV7JSJM3CADPG6EYGOS2HFQ, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.524262138 (21h9m13.563835258s) [noise=true , address=45.91.116.6:4973 , piece_id=X63L2GPVONW7ORT5DYOQIHV46EG4DIHEB6TLOZMXKLCQPR5EWDFA, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.524469726 (21h9m13.564042846s) [noise=true , address=45.91.116.6:4974 , piece_id=ZUB4DD6ZXILUSYESNMGNOCKZVTPUY24N52YF2KY3Z36JYNDGRM2A, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.524553666 (21h9m13.564126786s) [noise=true , address=45.91.116.4:4939 , piece_id=BXLXHF3W5CPAS73RZXC4GL4PPN5PRMVLSSW4ADSW2M2KWRXQY77Q, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.524760001 (21h9m13.564333121s) [noise=true , address=77.90.31.10:4519 , piece_id=5RGPSBJO5NDUDD7Y4NH5CYTGF2FB5RO6ADKEP7YUGZFVMCTACVKA, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.524839723 (21h9m13.564412843s) [noise=true , address=45.91.116.12:4225 , piece_id=JQOKRSTWFE56HRHV4BBEL3ZR7T24XBHVYYU4WA4SWJGF432EPOLQ, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.525132163 (21h9m13.564705293s) [noise=true , address=45.91.116.12:4227 , piece_id=7BZ46KAIEIJDIKHXRLDCMB4Y6E7KSC4QLBOLY2A2NZFA4GVVXD2A, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.525296006 (21h9m13.564869127s) [noise=true , address=45.91.116.6:4975 , piece_id=7U2D53SCSJ5GUGLFWEH6FA66Q5U7EIK7YVBTP5GP5P5OOAVOHQWA, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.525386890 (21h9m13.56496001s) [noise=true , address=45.91.116.10:4083 , piece_id=2XGLF7IS7LC7TU5X2M27UEYQBVX3UML3UEWSFUISJOXF6AVKITTQ, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.525611531 (21h9m13.565184651s) [noise=true , address=45.91.116.7:4009 , piece_id=TYKXIJX7XIMFYKEA5RMORUFVO7UDVKNVBSJAIFUEIB3A76G4HN3Q, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.525648361 (21h9m13.565221481s) [noise=true , address=45.91.116.14:4191 , piece_id=BLZ4EZD5DQ7QQNNRLNCKAB2CQBZGQ2SYKGJJIOZCXNGS4SYOFNXA, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.525870506 (21h9m13.565443626s) [noise=true , address=45.91.116.11:4117 , piece_id=5E2JPHN5T2SETJNOGGAESLRU7PRPUCUDEO25QDW6PA73EKVGMRYQ, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.525935922 (21h9m13.565509042s) [noise=true , address=45.91.116.7:4010 , piece_id=UQRAOTS6AILKBAR6QC3ZD3VATJEYM2T6T4K2KDVU3XRKYQSEB66A, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.526222790 (21h9m13.56579591s) [noise=true , address=45.91.116.13:4153 , piece_id=KUSBCYGRA2XFXZPOYJG4EYGZ7FOSKPKV2EEFQHUND6TPTNQX7AGQ, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.526424746 (21h9m13.565997866s) [noise=true , address=77.90.31.10:4518 , piece_id=XBJTZKIXGIW3WPKU4RWYRZBMAC67YYDYRNAJNBWRYX6SHU3T73WA, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:39.362042598 (21h9m15.401615708s) [upload=18635]: Next returned nil segment Sep 7 10:37:39.362059750 (21h9m15.40163287s) [upload=18635]: Waiting for error group managing segments... Sep 7 10:37:39.669685089 (21h9m15.709258209s) [noise=true , address=45.91.116.14:4191 , piece_id=BLZ4EZD5DQ7QQNNRLNCKAB2CQBZGQ2SYKGJJIOZCXNGS4SYOFNXA, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.679437981 (21h9m15.719011101s) [noise=true , address=77.90.31.10:4519 , piece_id=5RGPSBJO5NDUDD7Y4NH5CYTGF2FB5RO6ADKEP7YUGZFVMCTACVKA, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.679555526 (21h9m15.719128646s) [noise=true , address=45.91.116.12:4225 , piece_id=JQOKRSTWFE56HRHV4BBEL3ZR7T24XBHVYYU4WA4SWJGF432EPOLQ, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.687220604 (21h9m15.726793724s) [noise=true , address=45.91.116.7:4010 , piece_id=UQRAOTS6AILKBAR6QC3ZD3VATJEYM2T6T4K2KDVU3XRKYQSEB66A, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.711092863 (21h9m15.750665983s) [noise=true , address=45.91.116.4:4939 , piece_id=BXLXHF3W5CPAS73RZXC4GL4PPN5PRMVLSSW4ADSW2M2KWRXQY77Q, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.734404882 (21h9m15.773978002s) [noise=true , address=45.91.116.7:4009 , piece_id=TYKXIJX7XIMFYKEA5RMORUFVO7UDVKNVBSJAIFUEIB3A76G4HN3Q, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.743143673 (21h9m15.782716793s) [noise=true , address=45.91.116.11:4119 , piece_id=Y5TR5QSARCJ3HXQZN6OWO7NTLMSGCSLCOEZVJRSB3VZQQL7O57NA, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.748574906 (21h9m15.788148016s) [noise=true , address=45.91.116.6:4974 , piece_id=ZUB4DD6ZXILUSYESNMGNOCKZVTPUY24N52YF2KY3Z36JYNDGRM2A, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.754311293 (21h9m15.793884413s) [noise=true , address=45.91.116.13:4153 , piece_id=KUSBCYGRA2XFXZPOYJG4EYGZ7FOSKPKV2EEFQHUND6TPTNQX7AGQ, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.755384276 (21h9m15.794957396s) [noise=true , address=45.91.116.5:4298 , piece_id=T4KK43URSFHH2LDPBX4CN66G4KVBCWV7JSJM3CADPG6EYGOS2HFQ, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.758010662 (21h9m15.797583782s) [noise=true , address=77.90.31.10:4518 , piece_id=XBJTZKIXGIW3WPKU4RWYRZBMAC67YYDYRNAJNBWRYX6SHU3T73WA, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.774473184 (21h9m15.814046304s) [noise=true , address=45.91.116.6:4973 , piece_id=X63L2GPVONW7ORT5DYOQIHV46EG4DIHEB6TLOZMXKLCQPR5EWDFA, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.817437797 (21h9m15.857010917s) [noise=true , address=45.91.116.11:4117 , piece_id=5E2JPHN5T2SETJNOGGAESLRU7PRPUCUDEO25QDW6PA73EKVGMRYQ, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.817462975 (21h9m15.857036096s) [seg_pos={0 0}, upload=18635]: Segment reached optimal threshold of 13 pieces. Sep 7 10:37:39.817597944 (21h9m15.857171064s) [noise=true , address=45.91.116.6:4975 , piece_id=7U2D53SCSJ5GUGLFWEH6FA66Q5U7EIK7YVBTP5GP5P5OOAVOHQWA, seg_pos={0 0}, upload=18635]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: expected piece hash; context canceled; EOF Sep 7 10:37:39.817683578 (21h9m15.857256698s) [noise=true , address=45.91.116.10:4083 , piece_id=2XGLF7IS7LC7TU5X2M27UEYQBVX3UML3UEWSFUISJOXF6AVKITTQ, seg_pos={0 0}, upload=18635]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: expected piece hash; context canceled; EOF Sep 7 10:37:39.817775434 (21h9m15.857348554s) [noise=true , address=45.91.116.12:4227 , piece_id=7BZ46KAIEIJDIKHXRLDCMB4Y6E7KSC4QLBOLY2A2NZFA4GVVXD2A, seg_pos={0 0}, upload=18635]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH): protocol: expected piece hash; context canceled; EOF Sep 7 10:37:39.817852270 (21h9m15.8574254s) [seg_pos={0 0}, upload=18635]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 7 10:37:39.817955087 (21h9m15.857528217s) [upload=18635]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 7 10:37:39.817961028 (21h9m15.857534148s) [upload=18635]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 7 10:37:40.298833319 (21h9m16.338406429s) [upload=18635]: Done uploading segments... Sep 7 10:37:40.298849160 (21h9m16.33842228s) [upload=18635]: Upload finished. err: <nil> Duration: 6.862692025s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................X....................................] duration: 6.355526593s FAIL (2025-09-07 11:04:06 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Sep 7 10:37:37.259810434 (21h9m13.299383554s) [upload=18635]: Starting upload Sep 7 10:37:37.259869848 (21h9m13.299442968s) [upload=18635]: Done starting upload Sep 7 10:37:37.259991862 (21h9m13.299564982s) [upload=18635]: Uploading segments... Sep 7 10:37:37.260091512 (21h9m13.299664632s) [upload=18635]: Got next segment. Inline: false Sep 7 10:37:37.260124284 (21h9m13.299697405s) [upload=18635]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 7 10:37:37.260129676 (21h9m13.299702796s) [upload=18635]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 7 10:37:37.523765418 (21h9m13.563338538s) [seg_pos={0 0}, upload=18635]: Begin upload segment... Sep 7 10:37:37.523848977 (21h9m13.563422098s) [seg_pos={0 0}, upload=18635]: Done begin upload segment. Sep 7 10:37:37.523875308 (21h9m13.563448418s) [noise=true , address=45.91.116.11:4119 , piece_id=Y5TR5QSARCJ3HXQZN6OWO7NTLMSGCSLCOEZVJRSB3VZQQL7O57NA, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.523927959 (21h9m13.563501079s) [noise=true , address=45.91.116.5:4298 , piece_id=T4KK43URSFHH2LDPBX4CN66G4KVBCWV7JSJM3CADPG6EYGOS2HFQ, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.524262138 (21h9m13.563835258s) [noise=true , address=45.91.116.6:4973 , piece_id=X63L2GPVONW7ORT5DYOQIHV46EG4DIHEB6TLOZMXKLCQPR5EWDFA, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.524469726 (21h9m13.564042846s) [noise=true , address=45.91.116.6:4974 , piece_id=ZUB4DD6ZXILUSYESNMGNOCKZVTPUY24N52YF2KY3Z36JYNDGRM2A, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.524553666 (21h9m13.564126786s) [noise=true , address=45.91.116.4:4939 , piece_id=BXLXHF3W5CPAS73RZXC4GL4PPN5PRMVLSSW4ADSW2M2KWRXQY77Q, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.524760001 (21h9m13.564333121s) [noise=true , address=77.90.31.10:4519 , piece_id=5RGPSBJO5NDUDD7Y4NH5CYTGF2FB5RO6ADKEP7YUGZFVMCTACVKA, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.524839723 (21h9m13.564412843s) [noise=true , address=45.91.116.12:4225 , piece_id=JQOKRSTWFE56HRHV4BBEL3ZR7T24XBHVYYU4WA4SWJGF432EPOLQ, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.525132163 (21h9m13.564705293s) [noise=true , address=45.91.116.12:4227 , piece_id=7BZ46KAIEIJDIKHXRLDCMB4Y6E7KSC4QLBOLY2A2NZFA4GVVXD2A, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.525296006 (21h9m13.564869127s) [noise=true , address=45.91.116.6:4975 , piece_id=7U2D53SCSJ5GUGLFWEH6FA66Q5U7EIK7YVBTP5GP5P5OOAVOHQWA, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.525386890 (21h9m13.56496001s) [noise=true , address=45.91.116.10:4083 , piece_id=2XGLF7IS7LC7TU5X2M27UEYQBVX3UML3UEWSFUISJOXF6AVKITTQ, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.525611531 (21h9m13.565184651s) [noise=true , address=45.91.116.7:4009 , piece_id=TYKXIJX7XIMFYKEA5RMORUFVO7UDVKNVBSJAIFUEIB3A76G4HN3Q, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.525648361 (21h9m13.565221481s) [noise=true , address=45.91.116.14:4191 , piece_id=BLZ4EZD5DQ7QQNNRLNCKAB2CQBZGQ2SYKGJJIOZCXNGS4SYOFNXA, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.525870506 (21h9m13.565443626s) [noise=true , address=45.91.116.11:4117 , piece_id=5E2JPHN5T2SETJNOGGAESLRU7PRPUCUDEO25QDW6PA73EKVGMRYQ, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.525935922 (21h9m13.565509042s) [noise=true , address=45.91.116.7:4010 , piece_id=UQRAOTS6AILKBAR6QC3ZD3VATJEYM2T6T4K2KDVU3XRKYQSEB66A, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.526222790 (21h9m13.56579591s) [noise=true , address=45.91.116.13:4153 , piece_id=KUSBCYGRA2XFXZPOYJG4EYGZ7FOSKPKV2EEFQHUND6TPTNQX7AGQ, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.526424746 (21h9m13.565997866s) [noise=true , address=77.90.31.10:4518 , piece_id=XBJTZKIXGIW3WPKU4RWYRZBMAC67YYDYRNAJNBWRYX6SHU3T73WA, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:39.362042598 (21h9m15.401615708s) [upload=18635]: Next returned nil segment Sep 7 10:37:39.362059750 (21h9m15.40163287s) [upload=18635]: Waiting for error group managing segments... Sep 7 10:37:39.669685089 (21h9m15.709258209s) [noise=true , address=45.91.116.14:4191 , piece_id=BLZ4EZD5DQ7QQNNRLNCKAB2CQBZGQ2SYKGJJIOZCXNGS4SYOFNXA, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.679437981 (21h9m15.719011101s) [noise=true , address=77.90.31.10:4519 , piece_id=5RGPSBJO5NDUDD7Y4NH5CYTGF2FB5RO6ADKEP7YUGZFVMCTACVKA, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.679555526 (21h9m15.719128646s) [noise=true , address=45.91.116.12:4225 , piece_id=JQOKRSTWFE56HRHV4BBEL3ZR7T24XBHVYYU4WA4SWJGF432EPOLQ, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.687220604 (21h9m15.726793724s) [noise=true , address=45.91.116.7:4010 , piece_id=UQRAOTS6AILKBAR6QC3ZD3VATJEYM2T6T4K2KDVU3XRKYQSEB66A, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.711092863 (21h9m15.750665983s) [noise=true , address=45.91.116.4:4939 , piece_id=BXLXHF3W5CPAS73RZXC4GL4PPN5PRMVLSSW4ADSW2M2KWRXQY77Q, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.734404882 (21h9m15.773978002s) [noise=true , address=45.91.116.7:4009 , piece_id=TYKXIJX7XIMFYKEA5RMORUFVO7UDVKNVBSJAIFUEIB3A76G4HN3Q, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.743143673 (21h9m15.782716793s) [noise=true , address=45.91.116.11:4119 , piece_id=Y5TR5QSARCJ3HXQZN6OWO7NTLMSGCSLCOEZVJRSB3VZQQL7O57NA, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.748574906 (21h9m15.788148016s) [noise=true , address=45.91.116.6:4974 , piece_id=ZUB4DD6ZXILUSYESNMGNOCKZVTPUY24N52YF2KY3Z36JYNDGRM2A, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.754311293 (21h9m15.793884413s) [noise=true , address=45.91.116.13:4153 , piece_id=KUSBCYGRA2XFXZPOYJG4EYGZ7FOSKPKV2EEFQHUND6TPTNQX7AGQ, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.755384276 (21h9m15.794957396s) [noise=true , address=45.91.116.5:4298 , piece_id=T4KK43URSFHH2LDPBX4CN66G4KVBCWV7JSJM3CADPG6EYGOS2HFQ, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.758010662 (21h9m15.797583782s) [noise=true , address=77.90.31.10:4518 , piece_id=XBJTZKIXGIW3WPKU4RWYRZBMAC67YYDYRNAJNBWRYX6SHU3T73WA, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.774473184 (21h9m15.814046304s) [noise=true , address=45.91.116.6:4973 , piece_id=X63L2GPVONW7ORT5DYOQIHV46EG4DIHEB6TLOZMXKLCQPR5EWDFA, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.817437797 (21h9m15.857010917s) [noise=true , address=45.91.116.11:4117 , piece_id=5E2JPHN5T2SETJNOGGAESLRU7PRPUCUDEO25QDW6PA73EKVGMRYQ, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.817462975 (21h9m15.857036096s) [seg_pos={0 0}, upload=18635]: Segment reached optimal threshold of 13 pieces. Sep 7 10:37:39.817597944 (21h9m15.857171064s) [noise=true , address=45.91.116.6:4975 , piece_id=7U2D53SCSJ5GUGLFWEH6FA66Q5U7EIK7YVBTP5GP5P5OOAVOHQWA, seg_pos={0 0}, upload=18635]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: expected piece hash; context canceled; EOF Sep 7 10:37:39.817683578 (21h9m15.857256698s) [noise=true , address=45.91.116.10:4083 , piece_id=2XGLF7IS7LC7TU5X2M27UEYQBVX3UML3UEWSFUISJOXF6AVKITTQ, seg_pos={0 0}, upload=18635]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: expected piece hash; context canceled; EOF Sep 7 10:37:39.817775434 (21h9m15.857348554s) [noise=true , address=45.91.116.12:4227 , piece_id=7BZ46KAIEIJDIKHXRLDCMB4Y6E7KSC4QLBOLY2A2NZFA4GVVXD2A, seg_pos={0 0}, upload=18635]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH): protocol: expected piece hash; context canceled; EOF Sep 7 10:37:39.817852270 (21h9m15.8574254s) [seg_pos={0 0}, upload=18635]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 7 10:37:39.817955087 (21h9m15.857528217s) [upload=18635]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 7 10:37:39.817961028 (21h9m15.857534148s) [upload=18635]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 7 10:37:40.298833319 (21h9m16.338406429s) [upload=18635]: Done uploading segments... Sep 7 10:37:40.298849160 (21h9m16.33842228s) [upload=18635]: Upload finished. err: <nil> Duration: 6.355549046s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................X...................................] duration: 7.506882825s FAIL (2025-09-07 11:03:52 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Sep 7 10:37:37.259810434 (21h9m13.299383554s) [upload=18635]: Starting upload Sep 7 10:37:37.259869848 (21h9m13.299442968s) [upload=18635]: Done starting upload Sep 7 10:37:37.259991862 (21h9m13.299564982s) [upload=18635]: Uploading segments... Sep 7 10:37:37.260091512 (21h9m13.299664632s) [upload=18635]: Got next segment. Inline: false Sep 7 10:37:37.260124284 (21h9m13.299697405s) [upload=18635]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 7 10:37:37.260129676 (21h9m13.299702796s) [upload=18635]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 7 10:37:37.523765418 (21h9m13.563338538s) [seg_pos={0 0}, upload=18635]: Begin upload segment... Sep 7 10:37:37.523848977 (21h9m13.563422098s) [seg_pos={0 0}, upload=18635]: Done begin upload segment. Sep 7 10:37:37.523875308 (21h9m13.563448418s) [noise=true , address=45.91.116.11:4119 , piece_id=Y5TR5QSARCJ3HXQZN6OWO7NTLMSGCSLCOEZVJRSB3VZQQL7O57NA, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.523927959 (21h9m13.563501079s) [noise=true , address=45.91.116.5:4298 , piece_id=T4KK43URSFHH2LDPBX4CN66G4KVBCWV7JSJM3CADPG6EYGOS2HFQ, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.524262138 (21h9m13.563835258s) [noise=true , address=45.91.116.6:4973 , piece_id=X63L2GPVONW7ORT5DYOQIHV46EG4DIHEB6TLOZMXKLCQPR5EWDFA, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.524469726 (21h9m13.564042846s) [noise=true , address=45.91.116.6:4974 , piece_id=ZUB4DD6ZXILUSYESNMGNOCKZVTPUY24N52YF2KY3Z36JYNDGRM2A, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.524553666 (21h9m13.564126786s) [noise=true , address=45.91.116.4:4939 , piece_id=BXLXHF3W5CPAS73RZXC4GL4PPN5PRMVLSSW4ADSW2M2KWRXQY77Q, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.524760001 (21h9m13.564333121s) [noise=true , address=77.90.31.10:4519 , piece_id=5RGPSBJO5NDUDD7Y4NH5CYTGF2FB5RO6ADKEP7YUGZFVMCTACVKA, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.524839723 (21h9m13.564412843s) [noise=true , address=45.91.116.12:4225 , piece_id=JQOKRSTWFE56HRHV4BBEL3ZR7T24XBHVYYU4WA4SWJGF432EPOLQ, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.525132163 (21h9m13.564705293s) [noise=true , address=45.91.116.12:4227 , piece_id=7BZ46KAIEIJDIKHXRLDCMB4Y6E7KSC4QLBOLY2A2NZFA4GVVXD2A, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.525296006 (21h9m13.564869127s) [noise=true , address=45.91.116.6:4975 , piece_id=7U2D53SCSJ5GUGLFWEH6FA66Q5U7EIK7YVBTP5GP5P5OOAVOHQWA, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.525386890 (21h9m13.56496001s) [noise=true , address=45.91.116.10:4083 , piece_id=2XGLF7IS7LC7TU5X2M27UEYQBVX3UML3UEWSFUISJOXF6AVKITTQ, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.525611531 (21h9m13.565184651s) [noise=true , address=45.91.116.7:4009 , piece_id=TYKXIJX7XIMFYKEA5RMORUFVO7UDVKNVBSJAIFUEIB3A76G4HN3Q, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.525648361 (21h9m13.565221481s) [noise=true , address=45.91.116.14:4191 , piece_id=BLZ4EZD5DQ7QQNNRLNCKAB2CQBZGQ2SYKGJJIOZCXNGS4SYOFNXA, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.525870506 (21h9m13.565443626s) [noise=true , address=45.91.116.11:4117 , piece_id=5E2JPHN5T2SETJNOGGAESLRU7PRPUCUDEO25QDW6PA73EKVGMRYQ, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.525935922 (21h9m13.565509042s) [noise=true , address=45.91.116.7:4010 , piece_id=UQRAOTS6AILKBAR6QC3ZD3VATJEYM2T6T4K2KDVU3XRKYQSEB66A, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.526222790 (21h9m13.56579591s) [noise=true , address=45.91.116.13:4153 , piece_id=KUSBCYGRA2XFXZPOYJG4EYGZ7FOSKPKV2EEFQHUND6TPTNQX7AGQ, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.526424746 (21h9m13.565997866s) [noise=true , address=77.90.31.10:4518 , piece_id=XBJTZKIXGIW3WPKU4RWYRZBMAC67YYDYRNAJNBWRYX6SHU3T73WA, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:39.362042598 (21h9m15.401615708s) [upload=18635]: Next returned nil segment Sep 7 10:37:39.362059750 (21h9m15.40163287s) [upload=18635]: Waiting for error group managing segments... Sep 7 10:37:39.669685089 (21h9m15.709258209s) [noise=true , address=45.91.116.14:4191 , piece_id=BLZ4EZD5DQ7QQNNRLNCKAB2CQBZGQ2SYKGJJIOZCXNGS4SYOFNXA, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.679437981 (21h9m15.719011101s) [noise=true , address=77.90.31.10:4519 , piece_id=5RGPSBJO5NDUDD7Y4NH5CYTGF2FB5RO6ADKEP7YUGZFVMCTACVKA, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.679555526 (21h9m15.719128646s) [noise=true , address=45.91.116.12:4225 , piece_id=JQOKRSTWFE56HRHV4BBEL3ZR7T24XBHVYYU4WA4SWJGF432EPOLQ, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.687220604 (21h9m15.726793724s) [noise=true , address=45.91.116.7:4010 , piece_id=UQRAOTS6AILKBAR6QC3ZD3VATJEYM2T6T4K2KDVU3XRKYQSEB66A, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.711092863 (21h9m15.750665983s) [noise=true , address=45.91.116.4:4939 , piece_id=BXLXHF3W5CPAS73RZXC4GL4PPN5PRMVLSSW4ADSW2M2KWRXQY77Q, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.734404882 (21h9m15.773978002s) [noise=true , address=45.91.116.7:4009 , piece_id=TYKXIJX7XIMFYKEA5RMORUFVO7UDVKNVBSJAIFUEIB3A76G4HN3Q, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.743143673 (21h9m15.782716793s) [noise=true , address=45.91.116.11:4119 , piece_id=Y5TR5QSARCJ3HXQZN6OWO7NTLMSGCSLCOEZVJRSB3VZQQL7O57NA, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.748574906 (21h9m15.788148016s) [noise=true , address=45.91.116.6:4974 , piece_id=ZUB4DD6ZXILUSYESNMGNOCKZVTPUY24N52YF2KY3Z36JYNDGRM2A, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.754311293 (21h9m15.793884413s) [noise=true , address=45.91.116.13:4153 , piece_id=KUSBCYGRA2XFXZPOYJG4EYGZ7FOSKPKV2EEFQHUND6TPTNQX7AGQ, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.755384276 (21h9m15.794957396s) [noise=true , address=45.91.116.5:4298 , piece_id=T4KK43URSFHH2LDPBX4CN66G4KVBCWV7JSJM3CADPG6EYGOS2HFQ, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.758010662 (21h9m15.797583782s) [noise=true , address=77.90.31.10:4518 , piece_id=XBJTZKIXGIW3WPKU4RWYRZBMAC67YYDYRNAJNBWRYX6SHU3T73WA, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.774473184 (21h9m15.814046304s) [noise=true , address=45.91.116.6:4973 , piece_id=X63L2GPVONW7ORT5DYOQIHV46EG4DIHEB6TLOZMXKLCQPR5EWDFA, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.817437797 (21h9m15.857010917s) [noise=true , address=45.91.116.11:4117 , piece_id=5E2JPHN5T2SETJNOGGAESLRU7PRPUCUDEO25QDW6PA73EKVGMRYQ, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.817462975 (21h9m15.857036096s) [seg_pos={0 0}, upload=18635]: Segment reached optimal threshold of 13 pieces. Sep 7 10:37:39.817597944 (21h9m15.857171064s) [noise=true , address=45.91.116.6:4975 , piece_id=7U2D53SCSJ5GUGLFWEH6FA66Q5U7EIK7YVBTP5GP5P5OOAVOHQWA, seg_pos={0 0}, upload=18635]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: expected piece hash; context canceled; EOF Sep 7 10:37:39.817683578 (21h9m15.857256698s) [noise=true , address=45.91.116.10:4083 , piece_id=2XGLF7IS7LC7TU5X2M27UEYQBVX3UML3UEWSFUISJOXF6AVKITTQ, seg_pos={0 0}, upload=18635]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: expected piece hash; context canceled; EOF Sep 7 10:37:39.817775434 (21h9m15.857348554s) [noise=true , address=45.91.116.12:4227 , piece_id=7BZ46KAIEIJDIKHXRLDCMB4Y6E7KSC4QLBOLY2A2NZFA4GVVXD2A, seg_pos={0 0}, upload=18635]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH): protocol: expected piece hash; context canceled; EOF Sep 7 10:37:39.817852270 (21h9m15.8574254s) [seg_pos={0 0}, upload=18635]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 7 10:37:39.817955087 (21h9m15.857528217s) [upload=18635]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 7 10:37:39.817961028 (21h9m15.857534148s) [upload=18635]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 7 10:37:40.298833319 (21h9m16.338406429s) [upload=18635]: Done uploading segments... Sep 7 10:37:40.298849160 (21h9m16.33842228s) [upload=18635]: Upload finished. err: <nil> Duration: 7.506913141s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................X..................................] duration: 12.668689051s FAIL (2025-09-07 11:03:13 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Sep 7 10:37:37.259810434 (21h9m13.299383554s) [upload=18635]: Starting upload Sep 7 10:37:37.259869848 (21h9m13.299442968s) [upload=18635]: Done starting upload Sep 7 10:37:37.259991862 (21h9m13.299564982s) [upload=18635]: Uploading segments... Sep 7 10:37:37.260091512 (21h9m13.299664632s) [upload=18635]: Got next segment. Inline: false Sep 7 10:37:37.260124284 (21h9m13.299697405s) [upload=18635]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 7 10:37:37.260129676 (21h9m13.299702796s) [upload=18635]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 7 10:37:37.523765418 (21h9m13.563338538s) [seg_pos={0 0}, upload=18635]: Begin upload segment... Sep 7 10:37:37.523848977 (21h9m13.563422098s) [seg_pos={0 0}, upload=18635]: Done begin upload segment. Sep 7 10:37:37.523875308 (21h9m13.563448418s) [noise=true , address=45.91.116.11:4119 , piece_id=Y5TR5QSARCJ3HXQZN6OWO7NTLMSGCSLCOEZVJRSB3VZQQL7O57NA, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.523927959 (21h9m13.563501079s) [noise=true , address=45.91.116.5:4298 , piece_id=T4KK43URSFHH2LDPBX4CN66G4KVBCWV7JSJM3CADPG6EYGOS2HFQ, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.524262138 (21h9m13.563835258s) [noise=true , address=45.91.116.6:4973 , piece_id=X63L2GPVONW7ORT5DYOQIHV46EG4DIHEB6TLOZMXKLCQPR5EWDFA, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.524469726 (21h9m13.564042846s) [noise=true , address=45.91.116.6:4974 , piece_id=ZUB4DD6ZXILUSYESNMGNOCKZVTPUY24N52YF2KY3Z36JYNDGRM2A, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.524553666 (21h9m13.564126786s) [noise=true , address=45.91.116.4:4939 , piece_id=BXLXHF3W5CPAS73RZXC4GL4PPN5PRMVLSSW4ADSW2M2KWRXQY77Q, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.524760001 (21h9m13.564333121s) [noise=true , address=77.90.31.10:4519 , piece_id=5RGPSBJO5NDUDD7Y4NH5CYTGF2FB5RO6ADKEP7YUGZFVMCTACVKA, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.524839723 (21h9m13.564412843s) [noise=true , address=45.91.116.12:4225 , piece_id=JQOKRSTWFE56HRHV4BBEL3ZR7T24XBHVYYU4WA4SWJGF432EPOLQ, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.525132163 (21h9m13.564705293s) [noise=true , address=45.91.116.12:4227 , piece_id=7BZ46KAIEIJDIKHXRLDCMB4Y6E7KSC4QLBOLY2A2NZFA4GVVXD2A, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.525296006 (21h9m13.564869127s) [noise=true , address=45.91.116.6:4975 , piece_id=7U2D53SCSJ5GUGLFWEH6FA66Q5U7EIK7YVBTP5GP5P5OOAVOHQWA, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.525386890 (21h9m13.56496001s) [noise=true , address=45.91.116.10:4083 , piece_id=2XGLF7IS7LC7TU5X2M27UEYQBVX3UML3UEWSFUISJOXF6AVKITTQ, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.525611531 (21h9m13.565184651s) [noise=true , address=45.91.116.7:4009 , piece_id=TYKXIJX7XIMFYKEA5RMORUFVO7UDVKNVBSJAIFUEIB3A76G4HN3Q, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.525648361 (21h9m13.565221481s) [noise=true , address=45.91.116.14:4191 , piece_id=BLZ4EZD5DQ7QQNNRLNCKAB2CQBZGQ2SYKGJJIOZCXNGS4SYOFNXA, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.525870506 (21h9m13.565443626s) [noise=true , address=45.91.116.11:4117 , piece_id=5E2JPHN5T2SETJNOGGAESLRU7PRPUCUDEO25QDW6PA73EKVGMRYQ, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.525935922 (21h9m13.565509042s) [noise=true , address=45.91.116.7:4010 , piece_id=UQRAOTS6AILKBAR6QC3ZD3VATJEYM2T6T4K2KDVU3XRKYQSEB66A, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.526222790 (21h9m13.56579591s) [noise=true , address=45.91.116.13:4153 , piece_id=KUSBCYGRA2XFXZPOYJG4EYGZ7FOSKPKV2EEFQHUND6TPTNQX7AGQ, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.526424746 (21h9m13.565997866s) [noise=true , address=77.90.31.10:4518 , piece_id=XBJTZKIXGIW3WPKU4RWYRZBMAC67YYDYRNAJNBWRYX6SHU3T73WA, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:39.362042598 (21h9m15.401615708s) [upload=18635]: Next returned nil segment Sep 7 10:37:39.362059750 (21h9m15.40163287s) [upload=18635]: Waiting for error group managing segments... Sep 7 10:37:39.669685089 (21h9m15.709258209s) [noise=true , address=45.91.116.14:4191 , piece_id=BLZ4EZD5DQ7QQNNRLNCKAB2CQBZGQ2SYKGJJIOZCXNGS4SYOFNXA, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.679437981 (21h9m15.719011101s) [noise=true , address=77.90.31.10:4519 , piece_id=5RGPSBJO5NDUDD7Y4NH5CYTGF2FB5RO6ADKEP7YUGZFVMCTACVKA, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.679555526 (21h9m15.719128646s) [noise=true , address=45.91.116.12:4225 , piece_id=JQOKRSTWFE56HRHV4BBEL3ZR7T24XBHVYYU4WA4SWJGF432EPOLQ, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.687220604 (21h9m15.726793724s) [noise=true , address=45.91.116.7:4010 , piece_id=UQRAOTS6AILKBAR6QC3ZD3VATJEYM2T6T4K2KDVU3XRKYQSEB66A, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.711092863 (21h9m15.750665983s) [noise=true , address=45.91.116.4:4939 , piece_id=BXLXHF3W5CPAS73RZXC4GL4PPN5PRMVLSSW4ADSW2M2KWRXQY77Q, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.734404882 (21h9m15.773978002s) [noise=true , address=45.91.116.7:4009 , piece_id=TYKXIJX7XIMFYKEA5RMORUFVO7UDVKNVBSJAIFUEIB3A76G4HN3Q, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.743143673 (21h9m15.782716793s) [noise=true , address=45.91.116.11:4119 , piece_id=Y5TR5QSARCJ3HXQZN6OWO7NTLMSGCSLCOEZVJRSB3VZQQL7O57NA, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.748574906 (21h9m15.788148016s) [noise=true , address=45.91.116.6:4974 , piece_id=ZUB4DD6ZXILUSYESNMGNOCKZVTPUY24N52YF2KY3Z36JYNDGRM2A, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.754311293 (21h9m15.793884413s) [noise=true , address=45.91.116.13:4153 , piece_id=KUSBCYGRA2XFXZPOYJG4EYGZ7FOSKPKV2EEFQHUND6TPTNQX7AGQ, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.755384276 (21h9m15.794957396s) [noise=true , address=45.91.116.5:4298 , piece_id=T4KK43URSFHH2LDPBX4CN66G4KVBCWV7JSJM3CADPG6EYGOS2HFQ, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.758010662 (21h9m15.797583782s) [noise=true , address=77.90.31.10:4518 , piece_id=XBJTZKIXGIW3WPKU4RWYRZBMAC67YYDYRNAJNBWRYX6SHU3T73WA, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.774473184 (21h9m15.814046304s) [noise=true , address=45.91.116.6:4973 , piece_id=X63L2GPVONW7ORT5DYOQIHV46EG4DIHEB6TLOZMXKLCQPR5EWDFA, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.817437797 (21h9m15.857010917s) [noise=true , address=45.91.116.11:4117 , piece_id=5E2JPHN5T2SETJNOGGAESLRU7PRPUCUDEO25QDW6PA73EKVGMRYQ, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.817462975 (21h9m15.857036096s) [seg_pos={0 0}, upload=18635]: Segment reached optimal threshold of 13 pieces. Sep 7 10:37:39.817597944 (21h9m15.857171064s) [noise=true , address=45.91.116.6:4975 , piece_id=7U2D53SCSJ5GUGLFWEH6FA66Q5U7EIK7YVBTP5GP5P5OOAVOHQWA, seg_pos={0 0}, upload=18635]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: expected piece hash; context canceled; EOF Sep 7 10:37:39.817683578 (21h9m15.857256698s) [noise=true , address=45.91.116.10:4083 , piece_id=2XGLF7IS7LC7TU5X2M27UEYQBVX3UML3UEWSFUISJOXF6AVKITTQ, seg_pos={0 0}, upload=18635]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: expected piece hash; context canceled; EOF Sep 7 10:37:39.817775434 (21h9m15.857348554s) [noise=true , address=45.91.116.12:4227 , piece_id=7BZ46KAIEIJDIKHXRLDCMB4Y6E7KSC4QLBOLY2A2NZFA4GVVXD2A, seg_pos={0 0}, upload=18635]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH): protocol: expected piece hash; context canceled; EOF Sep 7 10:37:39.817852270 (21h9m15.8574254s) [seg_pos={0 0}, upload=18635]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 7 10:37:39.817955087 (21h9m15.857528217s) [upload=18635]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 7 10:37:39.817961028 (21h9m15.857534148s) [upload=18635]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 7 10:37:40.298833319 (21h9m16.338406429s) [upload=18635]: Done uploading segments... Sep 7 10:37:40.298849160 (21h9m16.33842228s) [upload=18635]: Upload finished. err: <nil> Duration: 12.668721203s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................X.................................] duration: 6.148545791s FAIL (2025-09-07 11:02:46 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Sep 7 10:37:37.259810434 (21h9m13.299383554s) [upload=18635]: Starting upload Sep 7 10:37:37.259869848 (21h9m13.299442968s) [upload=18635]: Done starting upload Sep 7 10:37:37.259991862 (21h9m13.299564982s) [upload=18635]: Uploading segments... Sep 7 10:37:37.260091512 (21h9m13.299664632s) [upload=18635]: Got next segment. Inline: false Sep 7 10:37:37.260124284 (21h9m13.299697405s) [upload=18635]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 7 10:37:37.260129676 (21h9m13.299702796s) [upload=18635]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 7 10:37:37.523765418 (21h9m13.563338538s) [seg_pos={0 0}, upload=18635]: Begin upload segment... Sep 7 10:37:37.523848977 (21h9m13.563422098s) [seg_pos={0 0}, upload=18635]: Done begin upload segment. Sep 7 10:37:37.523875308 (21h9m13.563448418s) [noise=true , address=45.91.116.11:4119 , piece_id=Y5TR5QSARCJ3HXQZN6OWO7NTLMSGCSLCOEZVJRSB3VZQQL7O57NA, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.523927959 (21h9m13.563501079s) [noise=true , address=45.91.116.5:4298 , piece_id=T4KK43URSFHH2LDPBX4CN66G4KVBCWV7JSJM3CADPG6EYGOS2HFQ, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.524262138 (21h9m13.563835258s) [noise=true , address=45.91.116.6:4973 , piece_id=X63L2GPVONW7ORT5DYOQIHV46EG4DIHEB6TLOZMXKLCQPR5EWDFA, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.524469726 (21h9m13.564042846s) [noise=true , address=45.91.116.6:4974 , piece_id=ZUB4DD6ZXILUSYESNMGNOCKZVTPUY24N52YF2KY3Z36JYNDGRM2A, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.524553666 (21h9m13.564126786s) [noise=true , address=45.91.116.4:4939 , piece_id=BXLXHF3W5CPAS73RZXC4GL4PPN5PRMVLSSW4ADSW2M2KWRXQY77Q, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.524760001 (21h9m13.564333121s) [noise=true , address=77.90.31.10:4519 , piece_id=5RGPSBJO5NDUDD7Y4NH5CYTGF2FB5RO6ADKEP7YUGZFVMCTACVKA, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.524839723 (21h9m13.564412843s) [noise=true , address=45.91.116.12:4225 , piece_id=JQOKRSTWFE56HRHV4BBEL3ZR7T24XBHVYYU4WA4SWJGF432EPOLQ, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.525132163 (21h9m13.564705293s) [noise=true , address=45.91.116.12:4227 , piece_id=7BZ46KAIEIJDIKHXRLDCMB4Y6E7KSC4QLBOLY2A2NZFA4GVVXD2A, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.525296006 (21h9m13.564869127s) [noise=true , address=45.91.116.6:4975 , piece_id=7U2D53SCSJ5GUGLFWEH6FA66Q5U7EIK7YVBTP5GP5P5OOAVOHQWA, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.525386890 (21h9m13.56496001s) [noise=true , address=45.91.116.10:4083 , piece_id=2XGLF7IS7LC7TU5X2M27UEYQBVX3UML3UEWSFUISJOXF6AVKITTQ, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.525611531 (21h9m13.565184651s) [noise=true , address=45.91.116.7:4009 , piece_id=TYKXIJX7XIMFYKEA5RMORUFVO7UDVKNVBSJAIFUEIB3A76G4HN3Q, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.525648361 (21h9m13.565221481s) [noise=true , address=45.91.116.14:4191 , piece_id=BLZ4EZD5DQ7QQNNRLNCKAB2CQBZGQ2SYKGJJIOZCXNGS4SYOFNXA, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.525870506 (21h9m13.565443626s) [noise=true , address=45.91.116.11:4117 , piece_id=5E2JPHN5T2SETJNOGGAESLRU7PRPUCUDEO25QDW6PA73EKVGMRYQ, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.525935922 (21h9m13.565509042s) [noise=true , address=45.91.116.7:4010 , piece_id=UQRAOTS6AILKBAR6QC3ZD3VATJEYM2T6T4K2KDVU3XRKYQSEB66A, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.526222790 (21h9m13.56579591s) [noise=true , address=45.91.116.13:4153 , piece_id=KUSBCYGRA2XFXZPOYJG4EYGZ7FOSKPKV2EEFQHUND6TPTNQX7AGQ, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.526424746 (21h9m13.565997866s) [noise=true , address=77.90.31.10:4518 , piece_id=XBJTZKIXGIW3WPKU4RWYRZBMAC67YYDYRNAJNBWRYX6SHU3T73WA, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:39.362042598 (21h9m15.401615708s) [upload=18635]: Next returned nil segment Sep 7 10:37:39.362059750 (21h9m15.40163287s) [upload=18635]: Waiting for error group managing segments... Sep 7 10:37:39.669685089 (21h9m15.709258209s) [noise=true , address=45.91.116.14:4191 , piece_id=BLZ4EZD5DQ7QQNNRLNCKAB2CQBZGQ2SYKGJJIOZCXNGS4SYOFNXA, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.679437981 (21h9m15.719011101s) [noise=true , address=77.90.31.10:4519 , piece_id=5RGPSBJO5NDUDD7Y4NH5CYTGF2FB5RO6ADKEP7YUGZFVMCTACVKA, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.679555526 (21h9m15.719128646s) [noise=true , address=45.91.116.12:4225 , piece_id=JQOKRSTWFE56HRHV4BBEL3ZR7T24XBHVYYU4WA4SWJGF432EPOLQ, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.687220604 (21h9m15.726793724s) [noise=true , address=45.91.116.7:4010 , piece_id=UQRAOTS6AILKBAR6QC3ZD3VATJEYM2T6T4K2KDVU3XRKYQSEB66A, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.711092863 (21h9m15.750665983s) [noise=true , address=45.91.116.4:4939 , piece_id=BXLXHF3W5CPAS73RZXC4GL4PPN5PRMVLSSW4ADSW2M2KWRXQY77Q, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.734404882 (21h9m15.773978002s) [noise=true , address=45.91.116.7:4009 , piece_id=TYKXIJX7XIMFYKEA5RMORUFVO7UDVKNVBSJAIFUEIB3A76G4HN3Q, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.743143673 (21h9m15.782716793s) [noise=true , address=45.91.116.11:4119 , piece_id=Y5TR5QSARCJ3HXQZN6OWO7NTLMSGCSLCOEZVJRSB3VZQQL7O57NA, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.748574906 (21h9m15.788148016s) [noise=true , address=45.91.116.6:4974 , piece_id=ZUB4DD6ZXILUSYESNMGNOCKZVTPUY24N52YF2KY3Z36JYNDGRM2A, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.754311293 (21h9m15.793884413s) [noise=true , address=45.91.116.13:4153 , piece_id=KUSBCYGRA2XFXZPOYJG4EYGZ7FOSKPKV2EEFQHUND6TPTNQX7AGQ, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.755384276 (21h9m15.794957396s) [noise=true , address=45.91.116.5:4298 , piece_id=T4KK43URSFHH2LDPBX4CN66G4KVBCWV7JSJM3CADPG6EYGOS2HFQ, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.758010662 (21h9m15.797583782s) [noise=true , address=77.90.31.10:4518 , piece_id=XBJTZKIXGIW3WPKU4RWYRZBMAC67YYDYRNAJNBWRYX6SHU3T73WA, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.774473184 (21h9m15.814046304s) [noise=true , address=45.91.116.6:4973 , piece_id=X63L2GPVONW7ORT5DYOQIHV46EG4DIHEB6TLOZMXKLCQPR5EWDFA, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.817437797 (21h9m15.857010917s) [noise=true , address=45.91.116.11:4117 , piece_id=5E2JPHN5T2SETJNOGGAESLRU7PRPUCUDEO25QDW6PA73EKVGMRYQ, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.817462975 (21h9m15.857036096s) [seg_pos={0 0}, upload=18635]: Segment reached optimal threshold of 13 pieces. Sep 7 10:37:39.817597944 (21h9m15.857171064s) [noise=true , address=45.91.116.6:4975 , piece_id=7U2D53SCSJ5GUGLFWEH6FA66Q5U7EIK7YVBTP5GP5P5OOAVOHQWA, seg_pos={0 0}, upload=18635]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: expected piece hash; context canceled; EOF Sep 7 10:37:39.817683578 (21h9m15.857256698s) [noise=true , address=45.91.116.10:4083 , piece_id=2XGLF7IS7LC7TU5X2M27UEYQBVX3UML3UEWSFUISJOXF6AVKITTQ, seg_pos={0 0}, upload=18635]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: expected piece hash; context canceled; EOF Sep 7 10:37:39.817775434 (21h9m15.857348554s) [noise=true , address=45.91.116.12:4227 , piece_id=7BZ46KAIEIJDIKHXRLDCMB4Y6E7KSC4QLBOLY2A2NZFA4GVVXD2A, seg_pos={0 0}, upload=18635]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH): protocol: expected piece hash; context canceled; EOF Sep 7 10:37:39.817852270 (21h9m15.8574254s) [seg_pos={0 0}, upload=18635]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 7 10:37:39.817955087 (21h9m15.857528217s) [upload=18635]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 7 10:37:39.817961028 (21h9m15.857534148s) [upload=18635]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 7 10:37:40.298833319 (21h9m16.338406429s) [upload=18635]: Done uploading segments... Sep 7 10:37:40.298849160 (21h9m16.33842228s) [upload=18635]: Upload finished. err: <nil> Duration: 6.148571399s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................X................................] duration: 6.733598643s FAIL (2025-09-07 11:02:21 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Sep 7 10:37:37.259810434 (21h9m13.299383554s) [upload=18635]: Starting upload Sep 7 10:37:37.259869848 (21h9m13.299442968s) [upload=18635]: Done starting upload Sep 7 10:37:37.259991862 (21h9m13.299564982s) [upload=18635]: Uploading segments... Sep 7 10:37:37.260091512 (21h9m13.299664632s) [upload=18635]: Got next segment. Inline: false Sep 7 10:37:37.260124284 (21h9m13.299697405s) [upload=18635]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 7 10:37:37.260129676 (21h9m13.299702796s) [upload=18635]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 7 10:37:37.523765418 (21h9m13.563338538s) [seg_pos={0 0}, upload=18635]: Begin upload segment... Sep 7 10:37:37.523848977 (21h9m13.563422098s) [seg_pos={0 0}, upload=18635]: Done begin upload segment. Sep 7 10:37:37.523875308 (21h9m13.563448418s) [noise=true , address=45.91.116.11:4119 , piece_id=Y5TR5QSARCJ3HXQZN6OWO7NTLMSGCSLCOEZVJRSB3VZQQL7O57NA, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.523927959 (21h9m13.563501079s) [noise=true , address=45.91.116.5:4298 , piece_id=T4KK43URSFHH2LDPBX4CN66G4KVBCWV7JSJM3CADPG6EYGOS2HFQ, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.524262138 (21h9m13.563835258s) [noise=true , address=45.91.116.6:4973 , piece_id=X63L2GPVONW7ORT5DYOQIHV46EG4DIHEB6TLOZMXKLCQPR5EWDFA, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.524469726 (21h9m13.564042846s) [noise=true , address=45.91.116.6:4974 , piece_id=ZUB4DD6ZXILUSYESNMGNOCKZVTPUY24N52YF2KY3Z36JYNDGRM2A, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.524553666 (21h9m13.564126786s) [noise=true , address=45.91.116.4:4939 , piece_id=BXLXHF3W5CPAS73RZXC4GL4PPN5PRMVLSSW4ADSW2M2KWRXQY77Q, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.524760001 (21h9m13.564333121s) [noise=true , address=77.90.31.10:4519 , piece_id=5RGPSBJO5NDUDD7Y4NH5CYTGF2FB5RO6ADKEP7YUGZFVMCTACVKA, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.524839723 (21h9m13.564412843s) [noise=true , address=45.91.116.12:4225 , piece_id=JQOKRSTWFE56HRHV4BBEL3ZR7T24XBHVYYU4WA4SWJGF432EPOLQ, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.525132163 (21h9m13.564705293s) [noise=true , address=45.91.116.12:4227 , piece_id=7BZ46KAIEIJDIKHXRLDCMB4Y6E7KSC4QLBOLY2A2NZFA4GVVXD2A, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.525296006 (21h9m13.564869127s) [noise=true , address=45.91.116.6:4975 , piece_id=7U2D53SCSJ5GUGLFWEH6FA66Q5U7EIK7YVBTP5GP5P5OOAVOHQWA, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.525386890 (21h9m13.56496001s) [noise=true , address=45.91.116.10:4083 , piece_id=2XGLF7IS7LC7TU5X2M27UEYQBVX3UML3UEWSFUISJOXF6AVKITTQ, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.525611531 (21h9m13.565184651s) [noise=true , address=45.91.116.7:4009 , piece_id=TYKXIJX7XIMFYKEA5RMORUFVO7UDVKNVBSJAIFUEIB3A76G4HN3Q, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.525648361 (21h9m13.565221481s) [noise=true , address=45.91.116.14:4191 , piece_id=BLZ4EZD5DQ7QQNNRLNCKAB2CQBZGQ2SYKGJJIOZCXNGS4SYOFNXA, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.525870506 (21h9m13.565443626s) [noise=true , address=45.91.116.11:4117 , piece_id=5E2JPHN5T2SETJNOGGAESLRU7PRPUCUDEO25QDW6PA73EKVGMRYQ, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.525935922 (21h9m13.565509042s) [noise=true , address=45.91.116.7:4010 , piece_id=UQRAOTS6AILKBAR6QC3ZD3VATJEYM2T6T4K2KDVU3XRKYQSEB66A, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.526222790 (21h9m13.56579591s) [noise=true , address=45.91.116.13:4153 , piece_id=KUSBCYGRA2XFXZPOYJG4EYGZ7FOSKPKV2EEFQHUND6TPTNQX7AGQ, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.526424746 (21h9m13.565997866s) [noise=true , address=77.90.31.10:4518 , piece_id=XBJTZKIXGIW3WPKU4RWYRZBMAC67YYDYRNAJNBWRYX6SHU3T73WA, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:39.362042598 (21h9m15.401615708s) [upload=18635]: Next returned nil segment Sep 7 10:37:39.362059750 (21h9m15.40163287s) [upload=18635]: Waiting for error group managing segments... Sep 7 10:37:39.669685089 (21h9m15.709258209s) [noise=true , address=45.91.116.14:4191 , piece_id=BLZ4EZD5DQ7QQNNRLNCKAB2CQBZGQ2SYKGJJIOZCXNGS4SYOFNXA, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.679437981 (21h9m15.719011101s) [noise=true , address=77.90.31.10:4519 , piece_id=5RGPSBJO5NDUDD7Y4NH5CYTGF2FB5RO6ADKEP7YUGZFVMCTACVKA, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.679555526 (21h9m15.719128646s) [noise=true , address=45.91.116.12:4225 , piece_id=JQOKRSTWFE56HRHV4BBEL3ZR7T24XBHVYYU4WA4SWJGF432EPOLQ, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.687220604 (21h9m15.726793724s) [noise=true , address=45.91.116.7:4010 , piece_id=UQRAOTS6AILKBAR6QC3ZD3VATJEYM2T6T4K2KDVU3XRKYQSEB66A, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.711092863 (21h9m15.750665983s) [noise=true , address=45.91.116.4:4939 , piece_id=BXLXHF3W5CPAS73RZXC4GL4PPN5PRMVLSSW4ADSW2M2KWRXQY77Q, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.734404882 (21h9m15.773978002s) [noise=true , address=45.91.116.7:4009 , piece_id=TYKXIJX7XIMFYKEA5RMORUFVO7UDVKNVBSJAIFUEIB3A76G4HN3Q, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.743143673 (21h9m15.782716793s) [noise=true , address=45.91.116.11:4119 , piece_id=Y5TR5QSARCJ3HXQZN6OWO7NTLMSGCSLCOEZVJRSB3VZQQL7O57NA, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.748574906 (21h9m15.788148016s) [noise=true , address=45.91.116.6:4974 , piece_id=ZUB4DD6ZXILUSYESNMGNOCKZVTPUY24N52YF2KY3Z36JYNDGRM2A, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.754311293 (21h9m15.793884413s) [noise=true , address=45.91.116.13:4153 , piece_id=KUSBCYGRA2XFXZPOYJG4EYGZ7FOSKPKV2EEFQHUND6TPTNQX7AGQ, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.755384276 (21h9m15.794957396s) [noise=true , address=45.91.116.5:4298 , piece_id=T4KK43URSFHH2LDPBX4CN66G4KVBCWV7JSJM3CADPG6EYGOS2HFQ, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.758010662 (21h9m15.797583782s) [noise=true , address=77.90.31.10:4518 , piece_id=XBJTZKIXGIW3WPKU4RWYRZBMAC67YYDYRNAJNBWRYX6SHU3T73WA, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.774473184 (21h9m15.814046304s) [noise=true , address=45.91.116.6:4973 , piece_id=X63L2GPVONW7ORT5DYOQIHV46EG4DIHEB6TLOZMXKLCQPR5EWDFA, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.817437797 (21h9m15.857010917s) [noise=true , address=45.91.116.11:4117 , piece_id=5E2JPHN5T2SETJNOGGAESLRU7PRPUCUDEO25QDW6PA73EKVGMRYQ, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.817462975 (21h9m15.857036096s) [seg_pos={0 0}, upload=18635]: Segment reached optimal threshold of 13 pieces. Sep 7 10:37:39.817597944 (21h9m15.857171064s) [noise=true , address=45.91.116.6:4975 , piece_id=7U2D53SCSJ5GUGLFWEH6FA66Q5U7EIK7YVBTP5GP5P5OOAVOHQWA, seg_pos={0 0}, upload=18635]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: expected piece hash; context canceled; EOF Sep 7 10:37:39.817683578 (21h9m15.857256698s) [noise=true , address=45.91.116.10:4083 , piece_id=2XGLF7IS7LC7TU5X2M27UEYQBVX3UML3UEWSFUISJOXF6AVKITTQ, seg_pos={0 0}, upload=18635]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: expected piece hash; context canceled; EOF Sep 7 10:37:39.817775434 (21h9m15.857348554s) [noise=true , address=45.91.116.12:4227 , piece_id=7BZ46KAIEIJDIKHXRLDCMB4Y6E7KSC4QLBOLY2A2NZFA4GVVXD2A, seg_pos={0 0}, upload=18635]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH): protocol: expected piece hash; context canceled; EOF Sep 7 10:37:39.817852270 (21h9m15.8574254s) [seg_pos={0 0}, upload=18635]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 7 10:37:39.817955087 (21h9m15.857528217s) [upload=18635]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 7 10:37:39.817961028 (21h9m15.857534148s) [upload=18635]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 7 10:37:40.298833319 (21h9m16.338406429s) [upload=18635]: Done uploading segments... Sep 7 10:37:40.298849160 (21h9m16.33842228s) [upload=18635]: Upload finished. err: <nil> Duration: 6.733631487s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................X...............................] duration: 7.897936426s FAIL (2025-09-07 11:01:48 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Sep 7 10:37:37.259810434 (21h9m13.299383554s) [upload=18635]: Starting upload Sep 7 10:37:37.259869848 (21h9m13.299442968s) [upload=18635]: Done starting upload Sep 7 10:37:37.259991862 (21h9m13.299564982s) [upload=18635]: Uploading segments... Sep 7 10:37:37.260091512 (21h9m13.299664632s) [upload=18635]: Got next segment. Inline: false Sep 7 10:37:37.260124284 (21h9m13.299697405s) [upload=18635]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 7 10:37:37.260129676 (21h9m13.299702796s) [upload=18635]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 7 10:37:37.523765418 (21h9m13.563338538s) [seg_pos={0 0}, upload=18635]: Begin upload segment... Sep 7 10:37:37.523848977 (21h9m13.563422098s) [seg_pos={0 0}, upload=18635]: Done begin upload segment. Sep 7 10:37:37.523875308 (21h9m13.563448418s) [noise=true , address=45.91.116.11:4119 , piece_id=Y5TR5QSARCJ3HXQZN6OWO7NTLMSGCSLCOEZVJRSB3VZQQL7O57NA, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.523927959 (21h9m13.563501079s) [noise=true , address=45.91.116.5:4298 , piece_id=T4KK43URSFHH2LDPBX4CN66G4KVBCWV7JSJM3CADPG6EYGOS2HFQ, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.524262138 (21h9m13.563835258s) [noise=true , address=45.91.116.6:4973 , piece_id=X63L2GPVONW7ORT5DYOQIHV46EG4DIHEB6TLOZMXKLCQPR5EWDFA, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.524469726 (21h9m13.564042846s) [noise=true , address=45.91.116.6:4974 , piece_id=ZUB4DD6ZXILUSYESNMGNOCKZVTPUY24N52YF2KY3Z36JYNDGRM2A, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.524553666 (21h9m13.564126786s) [noise=true , address=45.91.116.4:4939 , piece_id=BXLXHF3W5CPAS73RZXC4GL4PPN5PRMVLSSW4ADSW2M2KWRXQY77Q, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.524760001 (21h9m13.564333121s) [noise=true , address=77.90.31.10:4519 , piece_id=5RGPSBJO5NDUDD7Y4NH5CYTGF2FB5RO6ADKEP7YUGZFVMCTACVKA, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.524839723 (21h9m13.564412843s) [noise=true , address=45.91.116.12:4225 , piece_id=JQOKRSTWFE56HRHV4BBEL3ZR7T24XBHVYYU4WA4SWJGF432EPOLQ, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.525132163 (21h9m13.564705293s) [noise=true , address=45.91.116.12:4227 , piece_id=7BZ46KAIEIJDIKHXRLDCMB4Y6E7KSC4QLBOLY2A2NZFA4GVVXD2A, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.525296006 (21h9m13.564869127s) [noise=true , address=45.91.116.6:4975 , piece_id=7U2D53SCSJ5GUGLFWEH6FA66Q5U7EIK7YVBTP5GP5P5OOAVOHQWA, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.525386890 (21h9m13.56496001s) [noise=true , address=45.91.116.10:4083 , piece_id=2XGLF7IS7LC7TU5X2M27UEYQBVX3UML3UEWSFUISJOXF6AVKITTQ, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.525611531 (21h9m13.565184651s) [noise=true , address=45.91.116.7:4009 , piece_id=TYKXIJX7XIMFYKEA5RMORUFVO7UDVKNVBSJAIFUEIB3A76G4HN3Q, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.525648361 (21h9m13.565221481s) [noise=true , address=45.91.116.14:4191 , piece_id=BLZ4EZD5DQ7QQNNRLNCKAB2CQBZGQ2SYKGJJIOZCXNGS4SYOFNXA, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.525870506 (21h9m13.565443626s) [noise=true , address=45.91.116.11:4117 , piece_id=5E2JPHN5T2SETJNOGGAESLRU7PRPUCUDEO25QDW6PA73EKVGMRYQ, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.525935922 (21h9m13.565509042s) [noise=true , address=45.91.116.7:4010 , piece_id=UQRAOTS6AILKBAR6QC3ZD3VATJEYM2T6T4K2KDVU3XRKYQSEB66A, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.526222790 (21h9m13.56579591s) [noise=true , address=45.91.116.13:4153 , piece_id=KUSBCYGRA2XFXZPOYJG4EYGZ7FOSKPKV2EEFQHUND6TPTNQX7AGQ, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.526424746 (21h9m13.565997866s) [noise=true , address=77.90.31.10:4518 , piece_id=XBJTZKIXGIW3WPKU4RWYRZBMAC67YYDYRNAJNBWRYX6SHU3T73WA, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:39.362042598 (21h9m15.401615708s) [upload=18635]: Next returned nil segment Sep 7 10:37:39.362059750 (21h9m15.40163287s) [upload=18635]: Waiting for error group managing segments... Sep 7 10:37:39.669685089 (21h9m15.709258209s) [noise=true , address=45.91.116.14:4191 , piece_id=BLZ4EZD5DQ7QQNNRLNCKAB2CQBZGQ2SYKGJJIOZCXNGS4SYOFNXA, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.679437981 (21h9m15.719011101s) [noise=true , address=77.90.31.10:4519 , piece_id=5RGPSBJO5NDUDD7Y4NH5CYTGF2FB5RO6ADKEP7YUGZFVMCTACVKA, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.679555526 (21h9m15.719128646s) [noise=true , address=45.91.116.12:4225 , piece_id=JQOKRSTWFE56HRHV4BBEL3ZR7T24XBHVYYU4WA4SWJGF432EPOLQ, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.687220604 (21h9m15.726793724s) [noise=true , address=45.91.116.7:4010 , piece_id=UQRAOTS6AILKBAR6QC3ZD3VATJEYM2T6T4K2KDVU3XRKYQSEB66A, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.711092863 (21h9m15.750665983s) [noise=true , address=45.91.116.4:4939 , piece_id=BXLXHF3W5CPAS73RZXC4GL4PPN5PRMVLSSW4ADSW2M2KWRXQY77Q, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.734404882 (21h9m15.773978002s) [noise=true , address=45.91.116.7:4009 , piece_id=TYKXIJX7XIMFYKEA5RMORUFVO7UDVKNVBSJAIFUEIB3A76G4HN3Q, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.743143673 (21h9m15.782716793s) [noise=true , address=45.91.116.11:4119 , piece_id=Y5TR5QSARCJ3HXQZN6OWO7NTLMSGCSLCOEZVJRSB3VZQQL7O57NA, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.748574906 (21h9m15.788148016s) [noise=true , address=45.91.116.6:4974 , piece_id=ZUB4DD6ZXILUSYESNMGNOCKZVTPUY24N52YF2KY3Z36JYNDGRM2A, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.754311293 (21h9m15.793884413s) [noise=true , address=45.91.116.13:4153 , piece_id=KUSBCYGRA2XFXZPOYJG4EYGZ7FOSKPKV2EEFQHUND6TPTNQX7AGQ, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.755384276 (21h9m15.794957396s) [noise=true , address=45.91.116.5:4298 , piece_id=T4KK43URSFHH2LDPBX4CN66G4KVBCWV7JSJM3CADPG6EYGOS2HFQ, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.758010662 (21h9m15.797583782s) [noise=true , address=77.90.31.10:4518 , piece_id=XBJTZKIXGIW3WPKU4RWYRZBMAC67YYDYRNAJNBWRYX6SHU3T73WA, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.774473184 (21h9m15.814046304s) [noise=true , address=45.91.116.6:4973 , piece_id=X63L2GPVONW7ORT5DYOQIHV46EG4DIHEB6TLOZMXKLCQPR5EWDFA, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.817437797 (21h9m15.857010917s) [noise=true , address=45.91.116.11:4117 , piece_id=5E2JPHN5T2SETJNOGGAESLRU7PRPUCUDEO25QDW6PA73EKVGMRYQ, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.817462975 (21h9m15.857036096s) [seg_pos={0 0}, upload=18635]: Segment reached optimal threshold of 13 pieces. Sep 7 10:37:39.817597944 (21h9m15.857171064s) [noise=true , address=45.91.116.6:4975 , piece_id=7U2D53SCSJ5GUGLFWEH6FA66Q5U7EIK7YVBTP5GP5P5OOAVOHQWA, seg_pos={0 0}, upload=18635]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: expected piece hash; context canceled; EOF Sep 7 10:37:39.817683578 (21h9m15.857256698s) [noise=true , address=45.91.116.10:4083 , piece_id=2XGLF7IS7LC7TU5X2M27UEYQBVX3UML3UEWSFUISJOXF6AVKITTQ, seg_pos={0 0}, upload=18635]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: expected piece hash; context canceled; EOF Sep 7 10:37:39.817775434 (21h9m15.857348554s) [noise=true , address=45.91.116.12:4227 , piece_id=7BZ46KAIEIJDIKHXRLDCMB4Y6E7KSC4QLBOLY2A2NZFA4GVVXD2A, seg_pos={0 0}, upload=18635]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH): protocol: expected piece hash; context canceled; EOF Sep 7 10:37:39.817852270 (21h9m15.8574254s) [seg_pos={0 0}, upload=18635]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 7 10:37:39.817955087 (21h9m15.857528217s) [upload=18635]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 7 10:37:39.817961028 (21h9m15.857534148s) [upload=18635]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 7 10:37:40.298833319 (21h9m16.338406429s) [upload=18635]: Done uploading segments... Sep 7 10:37:40.298849160 (21h9m16.33842228s) [upload=18635]: Upload finished. err: <nil> Duration: 7.897966533s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................X..............................] duration: 25.421370386s FAIL (2025-09-07 11:01:25 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Sep 7 10:37:37.259810434 (21h9m13.299383554s) [upload=18635]: Starting upload Sep 7 10:37:37.259869848 (21h9m13.299442968s) [upload=18635]: Done starting upload Sep 7 10:37:37.259991862 (21h9m13.299564982s) [upload=18635]: Uploading segments... Sep 7 10:37:37.260091512 (21h9m13.299664632s) [upload=18635]: Got next segment. Inline: false Sep 7 10:37:37.260124284 (21h9m13.299697405s) [upload=18635]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 7 10:37:37.260129676 (21h9m13.299702796s) [upload=18635]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 7 10:37:37.523765418 (21h9m13.563338538s) [seg_pos={0 0}, upload=18635]: Begin upload segment... Sep 7 10:37:37.523848977 (21h9m13.563422098s) [seg_pos={0 0}, upload=18635]: Done begin upload segment. Sep 7 10:37:37.523875308 (21h9m13.563448418s) [noise=true , address=45.91.116.11:4119 , piece_id=Y5TR5QSARCJ3HXQZN6OWO7NTLMSGCSLCOEZVJRSB3VZQQL7O57NA, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.523927959 (21h9m13.563501079s) [noise=true , address=45.91.116.5:4298 , piece_id=T4KK43URSFHH2LDPBX4CN66G4KVBCWV7JSJM3CADPG6EYGOS2HFQ, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.524262138 (21h9m13.563835258s) [noise=true , address=45.91.116.6:4973 , piece_id=X63L2GPVONW7ORT5DYOQIHV46EG4DIHEB6TLOZMXKLCQPR5EWDFA, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.524469726 (21h9m13.564042846s) [noise=true , address=45.91.116.6:4974 , piece_id=ZUB4DD6ZXILUSYESNMGNOCKZVTPUY24N52YF2KY3Z36JYNDGRM2A, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.524553666 (21h9m13.564126786s) [noise=true , address=45.91.116.4:4939 , piece_id=BXLXHF3W5CPAS73RZXC4GL4PPN5PRMVLSSW4ADSW2M2KWRXQY77Q, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.524760001 (21h9m13.564333121s) [noise=true , address=77.90.31.10:4519 , piece_id=5RGPSBJO5NDUDD7Y4NH5CYTGF2FB5RO6ADKEP7YUGZFVMCTACVKA, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.524839723 (21h9m13.564412843s) [noise=true , address=45.91.116.12:4225 , piece_id=JQOKRSTWFE56HRHV4BBEL3ZR7T24XBHVYYU4WA4SWJGF432EPOLQ, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.525132163 (21h9m13.564705293s) [noise=true , address=45.91.116.12:4227 , piece_id=7BZ46KAIEIJDIKHXRLDCMB4Y6E7KSC4QLBOLY2A2NZFA4GVVXD2A, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.525296006 (21h9m13.564869127s) [noise=true , address=45.91.116.6:4975 , piece_id=7U2D53SCSJ5GUGLFWEH6FA66Q5U7EIK7YVBTP5GP5P5OOAVOHQWA, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.525386890 (21h9m13.56496001s) [noise=true , address=45.91.116.10:4083 , piece_id=2XGLF7IS7LC7TU5X2M27UEYQBVX3UML3UEWSFUISJOXF6AVKITTQ, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.525611531 (21h9m13.565184651s) [noise=true , address=45.91.116.7:4009 , piece_id=TYKXIJX7XIMFYKEA5RMORUFVO7UDVKNVBSJAIFUEIB3A76G4HN3Q, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.525648361 (21h9m13.565221481s) [noise=true , address=45.91.116.14:4191 , piece_id=BLZ4EZD5DQ7QQNNRLNCKAB2CQBZGQ2SYKGJJIOZCXNGS4SYOFNXA, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.525870506 (21h9m13.565443626s) [noise=true , address=45.91.116.11:4117 , piece_id=5E2JPHN5T2SETJNOGGAESLRU7PRPUCUDEO25QDW6PA73EKVGMRYQ, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.525935922 (21h9m13.565509042s) [noise=true , address=45.91.116.7:4010 , piece_id=UQRAOTS6AILKBAR6QC3ZD3VATJEYM2T6T4K2KDVU3XRKYQSEB66A, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.526222790 (21h9m13.56579591s) [noise=true , address=45.91.116.13:4153 , piece_id=KUSBCYGRA2XFXZPOYJG4EYGZ7FOSKPKV2EEFQHUND6TPTNQX7AGQ, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.526424746 (21h9m13.565997866s) [noise=true , address=77.90.31.10:4518 , piece_id=XBJTZKIXGIW3WPKU4RWYRZBMAC67YYDYRNAJNBWRYX6SHU3T73WA, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:39.362042598 (21h9m15.401615708s) [upload=18635]: Next returned nil segment Sep 7 10:37:39.362059750 (21h9m15.40163287s) [upload=18635]: Waiting for error group managing segments... Sep 7 10:37:39.669685089 (21h9m15.709258209s) [noise=true , address=45.91.116.14:4191 , piece_id=BLZ4EZD5DQ7QQNNRLNCKAB2CQBZGQ2SYKGJJIOZCXNGS4SYOFNXA, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.679437981 (21h9m15.719011101s) [noise=true , address=77.90.31.10:4519 , piece_id=5RGPSBJO5NDUDD7Y4NH5CYTGF2FB5RO6ADKEP7YUGZFVMCTACVKA, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.679555526 (21h9m15.719128646s) [noise=true , address=45.91.116.12:4225 , piece_id=JQOKRSTWFE56HRHV4BBEL3ZR7T24XBHVYYU4WA4SWJGF432EPOLQ, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.687220604 (21h9m15.726793724s) [noise=true , address=45.91.116.7:4010 , piece_id=UQRAOTS6AILKBAR6QC3ZD3VATJEYM2T6T4K2KDVU3XRKYQSEB66A, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.711092863 (21h9m15.750665983s) [noise=true , address=45.91.116.4:4939 , piece_id=BXLXHF3W5CPAS73RZXC4GL4PPN5PRMVLSSW4ADSW2M2KWRXQY77Q, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.734404882 (21h9m15.773978002s) [noise=true , address=45.91.116.7:4009 , piece_id=TYKXIJX7XIMFYKEA5RMORUFVO7UDVKNVBSJAIFUEIB3A76G4HN3Q, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.743143673 (21h9m15.782716793s) [noise=true , address=45.91.116.11:4119 , piece_id=Y5TR5QSARCJ3HXQZN6OWO7NTLMSGCSLCOEZVJRSB3VZQQL7O57NA, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.748574906 (21h9m15.788148016s) [noise=true , address=45.91.116.6:4974 , piece_id=ZUB4DD6ZXILUSYESNMGNOCKZVTPUY24N52YF2KY3Z36JYNDGRM2A, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.754311293 (21h9m15.793884413s) [noise=true , address=45.91.116.13:4153 , piece_id=KUSBCYGRA2XFXZPOYJG4EYGZ7FOSKPKV2EEFQHUND6TPTNQX7AGQ, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.755384276 (21h9m15.794957396s) [noise=true , address=45.91.116.5:4298 , piece_id=T4KK43URSFHH2LDPBX4CN66G4KVBCWV7JSJM3CADPG6EYGOS2HFQ, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.758010662 (21h9m15.797583782s) [noise=true , address=77.90.31.10:4518 , piece_id=XBJTZKIXGIW3WPKU4RWYRZBMAC67YYDYRNAJNBWRYX6SHU3T73WA, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.774473184 (21h9m15.814046304s) [noise=true , address=45.91.116.6:4973 , piece_id=X63L2GPVONW7ORT5DYOQIHV46EG4DIHEB6TLOZMXKLCQPR5EWDFA, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.817437797 (21h9m15.857010917s) [noise=true , address=45.91.116.11:4117 , piece_id=5E2JPHN5T2SETJNOGGAESLRU7PRPUCUDEO25QDW6PA73EKVGMRYQ, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.817462975 (21h9m15.857036096s) [seg_pos={0 0}, upload=18635]: Segment reached optimal threshold of 13 pieces. Sep 7 10:37:39.817597944 (21h9m15.857171064s) [noise=true , address=45.91.116.6:4975 , piece_id=7U2D53SCSJ5GUGLFWEH6FA66Q5U7EIK7YVBTP5GP5P5OOAVOHQWA, seg_pos={0 0}, upload=18635]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: expected piece hash; context canceled; EOF Sep 7 10:37:39.817683578 (21h9m15.857256698s) [noise=true , address=45.91.116.10:4083 , piece_id=2XGLF7IS7LC7TU5X2M27UEYQBVX3UML3UEWSFUISJOXF6AVKITTQ, seg_pos={0 0}, upload=18635]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: expected piece hash; context canceled; EOF Sep 7 10:37:39.817775434 (21h9m15.857348554s) [noise=true , address=45.91.116.12:4227 , piece_id=7BZ46KAIEIJDIKHXRLDCMB4Y6E7KSC4QLBOLY2A2NZFA4GVVXD2A, seg_pos={0 0}, upload=18635]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH): protocol: expected piece hash; context canceled; EOF Sep 7 10:37:39.817852270 (21h9m15.8574254s) [seg_pos={0 0}, upload=18635]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 7 10:37:39.817955087 (21h9m15.857528217s) [upload=18635]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 7 10:37:39.817961028 (21h9m15.857534148s) [upload=18635]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 7 10:37:40.298833319 (21h9m16.338406429s) [upload=18635]: Done uploading segments... Sep 7 10:37:40.298849160 (21h9m16.33842228s) [upload=18635]: Upload finished. err: <nil> Duration: 25.421412185s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................X.............................] duration: 6.078362353s FAIL (2025-09-07 11:00:23 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Sep 7 10:37:37.259810434 (21h9m13.299383554s) [upload=18635]: Starting upload Sep 7 10:37:37.259869848 (21h9m13.299442968s) [upload=18635]: Done starting upload Sep 7 10:37:37.259991862 (21h9m13.299564982s) [upload=18635]: Uploading segments... Sep 7 10:37:37.260091512 (21h9m13.299664632s) [upload=18635]: Got next segment. Inline: false Sep 7 10:37:37.260124284 (21h9m13.299697405s) [upload=18635]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 7 10:37:37.260129676 (21h9m13.299702796s) [upload=18635]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 7 10:37:37.523765418 (21h9m13.563338538s) [seg_pos={0 0}, upload=18635]: Begin upload segment... Sep 7 10:37:37.523848977 (21h9m13.563422098s) [seg_pos={0 0}, upload=18635]: Done begin upload segment. Sep 7 10:37:37.523875308 (21h9m13.563448418s) [noise=true , address=45.91.116.11:4119 , piece_id=Y5TR5QSARCJ3HXQZN6OWO7NTLMSGCSLCOEZVJRSB3VZQQL7O57NA, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.523927959 (21h9m13.563501079s) [noise=true , address=45.91.116.5:4298 , piece_id=T4KK43URSFHH2LDPBX4CN66G4KVBCWV7JSJM3CADPG6EYGOS2HFQ, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.524262138 (21h9m13.563835258s) [noise=true , address=45.91.116.6:4973 , piece_id=X63L2GPVONW7ORT5DYOQIHV46EG4DIHEB6TLOZMXKLCQPR5EWDFA, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.524469726 (21h9m13.564042846s) [noise=true , address=45.91.116.6:4974 , piece_id=ZUB4DD6ZXILUSYESNMGNOCKZVTPUY24N52YF2KY3Z36JYNDGRM2A, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.524553666 (21h9m13.564126786s) [noise=true , address=45.91.116.4:4939 , piece_id=BXLXHF3W5CPAS73RZXC4GL4PPN5PRMVLSSW4ADSW2M2KWRXQY77Q, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.524760001 (21h9m13.564333121s) [noise=true , address=77.90.31.10:4519 , piece_id=5RGPSBJO5NDUDD7Y4NH5CYTGF2FB5RO6ADKEP7YUGZFVMCTACVKA, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.524839723 (21h9m13.564412843s) [noise=true , address=45.91.116.12:4225 , piece_id=JQOKRSTWFE56HRHV4BBEL3ZR7T24XBHVYYU4WA4SWJGF432EPOLQ, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.525132163 (21h9m13.564705293s) [noise=true , address=45.91.116.12:4227 , piece_id=7BZ46KAIEIJDIKHXRLDCMB4Y6E7KSC4QLBOLY2A2NZFA4GVVXD2A, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.525296006 (21h9m13.564869127s) [noise=true , address=45.91.116.6:4975 , piece_id=7U2D53SCSJ5GUGLFWEH6FA66Q5U7EIK7YVBTP5GP5P5OOAVOHQWA, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.525386890 (21h9m13.56496001s) [noise=true , address=45.91.116.10:4083 , piece_id=2XGLF7IS7LC7TU5X2M27UEYQBVX3UML3UEWSFUISJOXF6AVKITTQ, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.525611531 (21h9m13.565184651s) [noise=true , address=45.91.116.7:4009 , piece_id=TYKXIJX7XIMFYKEA5RMORUFVO7UDVKNVBSJAIFUEIB3A76G4HN3Q, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.525648361 (21h9m13.565221481s) [noise=true , address=45.91.116.14:4191 , piece_id=BLZ4EZD5DQ7QQNNRLNCKAB2CQBZGQ2SYKGJJIOZCXNGS4SYOFNXA, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.525870506 (21h9m13.565443626s) [noise=true , address=45.91.116.11:4117 , piece_id=5E2JPHN5T2SETJNOGGAESLRU7PRPUCUDEO25QDW6PA73EKVGMRYQ, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.525935922 (21h9m13.565509042s) [noise=true , address=45.91.116.7:4010 , piece_id=UQRAOTS6AILKBAR6QC3ZD3VATJEYM2T6T4K2KDVU3XRKYQSEB66A, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.526222790 (21h9m13.56579591s) [noise=true , address=45.91.116.13:4153 , piece_id=KUSBCYGRA2XFXZPOYJG4EYGZ7FOSKPKV2EEFQHUND6TPTNQX7AGQ, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.526424746 (21h9m13.565997866s) [noise=true , address=77.90.31.10:4518 , piece_id=XBJTZKIXGIW3WPKU4RWYRZBMAC67YYDYRNAJNBWRYX6SHU3T73WA, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:39.362042598 (21h9m15.401615708s) [upload=18635]: Next returned nil segment Sep 7 10:37:39.362059750 (21h9m15.40163287s) [upload=18635]: Waiting for error group managing segments... Sep 7 10:37:39.669685089 (21h9m15.709258209s) [noise=true , address=45.91.116.14:4191 , piece_id=BLZ4EZD5DQ7QQNNRLNCKAB2CQBZGQ2SYKGJJIOZCXNGS4SYOFNXA, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.679437981 (21h9m15.719011101s) [noise=true , address=77.90.31.10:4519 , piece_id=5RGPSBJO5NDUDD7Y4NH5CYTGF2FB5RO6ADKEP7YUGZFVMCTACVKA, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.679555526 (21h9m15.719128646s) [noise=true , address=45.91.116.12:4225 , piece_id=JQOKRSTWFE56HRHV4BBEL3ZR7T24XBHVYYU4WA4SWJGF432EPOLQ, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.687220604 (21h9m15.726793724s) [noise=true , address=45.91.116.7:4010 , piece_id=UQRAOTS6AILKBAR6QC3ZD3VATJEYM2T6T4K2KDVU3XRKYQSEB66A, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.711092863 (21h9m15.750665983s) [noise=true , address=45.91.116.4:4939 , piece_id=BXLXHF3W5CPAS73RZXC4GL4PPN5PRMVLSSW4ADSW2M2KWRXQY77Q, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.734404882 (21h9m15.773978002s) [noise=true , address=45.91.116.7:4009 , piece_id=TYKXIJX7XIMFYKEA5RMORUFVO7UDVKNVBSJAIFUEIB3A76G4HN3Q, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.743143673 (21h9m15.782716793s) [noise=true , address=45.91.116.11:4119 , piece_id=Y5TR5QSARCJ3HXQZN6OWO7NTLMSGCSLCOEZVJRSB3VZQQL7O57NA, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.748574906 (21h9m15.788148016s) [noise=true , address=45.91.116.6:4974 , piece_id=ZUB4DD6ZXILUSYESNMGNOCKZVTPUY24N52YF2KY3Z36JYNDGRM2A, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.754311293 (21h9m15.793884413s) [noise=true , address=45.91.116.13:4153 , piece_id=KUSBCYGRA2XFXZPOYJG4EYGZ7FOSKPKV2EEFQHUND6TPTNQX7AGQ, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.755384276 (21h9m15.794957396s) [noise=true , address=45.91.116.5:4298 , piece_id=T4KK43URSFHH2LDPBX4CN66G4KVBCWV7JSJM3CADPG6EYGOS2HFQ, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.758010662 (21h9m15.797583782s) [noise=true , address=77.90.31.10:4518 , piece_id=XBJTZKIXGIW3WPKU4RWYRZBMAC67YYDYRNAJNBWRYX6SHU3T73WA, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.774473184 (21h9m15.814046304s) [noise=true , address=45.91.116.6:4973 , piece_id=X63L2GPVONW7ORT5DYOQIHV46EG4DIHEB6TLOZMXKLCQPR5EWDFA, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.817437797 (21h9m15.857010917s) [noise=true , address=45.91.116.11:4117 , piece_id=5E2JPHN5T2SETJNOGGAESLRU7PRPUCUDEO25QDW6PA73EKVGMRYQ, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.817462975 (21h9m15.857036096s) [seg_pos={0 0}, upload=18635]: Segment reached optimal threshold of 13 pieces. Sep 7 10:37:39.817597944 (21h9m15.857171064s) [noise=true , address=45.91.116.6:4975 , piece_id=7U2D53SCSJ5GUGLFWEH6FA66Q5U7EIK7YVBTP5GP5P5OOAVOHQWA, seg_pos={0 0}, upload=18635]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: expected piece hash; context canceled; EOF Sep 7 10:37:39.817683578 (21h9m15.857256698s) [noise=true , address=45.91.116.10:4083 , piece_id=2XGLF7IS7LC7TU5X2M27UEYQBVX3UML3UEWSFUISJOXF6AVKITTQ, seg_pos={0 0}, upload=18635]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: expected piece hash; context canceled; EOF Sep 7 10:37:39.817775434 (21h9m15.857348554s) [noise=true , address=45.91.116.12:4227 , piece_id=7BZ46KAIEIJDIKHXRLDCMB4Y6E7KSC4QLBOLY2A2NZFA4GVVXD2A, seg_pos={0 0}, upload=18635]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH): protocol: expected piece hash; context canceled; EOF Sep 7 10:37:39.817852270 (21h9m15.8574254s) [seg_pos={0 0}, upload=18635]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 7 10:37:39.817955087 (21h9m15.857528217s) [upload=18635]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 7 10:37:39.817961028 (21h9m15.857534148s) [upload=18635]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 7 10:37:40.298833319 (21h9m16.338406429s) [upload=18635]: Done uploading segments... Sep 7 10:37:40.298849160 (21h9m16.33842228s) [upload=18635]: Upload finished. err: <nil> Duration: 6.078398431s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................X............................] duration: 7.388213174s FAIL (2025-09-07 11:00:01 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Sep 7 10:37:37.259810434 (21h9m13.299383554s) [upload=18635]: Starting upload Sep 7 10:37:37.259869848 (21h9m13.299442968s) [upload=18635]: Done starting upload Sep 7 10:37:37.259991862 (21h9m13.299564982s) [upload=18635]: Uploading segments... Sep 7 10:37:37.260091512 (21h9m13.299664632s) [upload=18635]: Got next segment. Inline: false Sep 7 10:37:37.260124284 (21h9m13.299697405s) [upload=18635]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 7 10:37:37.260129676 (21h9m13.299702796s) [upload=18635]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 7 10:37:37.523765418 (21h9m13.563338538s) [seg_pos={0 0}, upload=18635]: Begin upload segment... Sep 7 10:37:37.523848977 (21h9m13.563422098s) [seg_pos={0 0}, upload=18635]: Done begin upload segment. Sep 7 10:37:37.523875308 (21h9m13.563448418s) [noise=true , address=45.91.116.11:4119 , piece_id=Y5TR5QSARCJ3HXQZN6OWO7NTLMSGCSLCOEZVJRSB3VZQQL7O57NA, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.523927959 (21h9m13.563501079s) [noise=true , address=45.91.116.5:4298 , piece_id=T4KK43URSFHH2LDPBX4CN66G4KVBCWV7JSJM3CADPG6EYGOS2HFQ, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.524262138 (21h9m13.563835258s) [noise=true , address=45.91.116.6:4973 , piece_id=X63L2GPVONW7ORT5DYOQIHV46EG4DIHEB6TLOZMXKLCQPR5EWDFA, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.524469726 (21h9m13.564042846s) [noise=true , address=45.91.116.6:4974 , piece_id=ZUB4DD6ZXILUSYESNMGNOCKZVTPUY24N52YF2KY3Z36JYNDGRM2A, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.524553666 (21h9m13.564126786s) [noise=true , address=45.91.116.4:4939 , piece_id=BXLXHF3W5CPAS73RZXC4GL4PPN5PRMVLSSW4ADSW2M2KWRXQY77Q, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.524760001 (21h9m13.564333121s) [noise=true , address=77.90.31.10:4519 , piece_id=5RGPSBJO5NDUDD7Y4NH5CYTGF2FB5RO6ADKEP7YUGZFVMCTACVKA, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.524839723 (21h9m13.564412843s) [noise=true , address=45.91.116.12:4225 , piece_id=JQOKRSTWFE56HRHV4BBEL3ZR7T24XBHVYYU4WA4SWJGF432EPOLQ, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.525132163 (21h9m13.564705293s) [noise=true , address=45.91.116.12:4227 , piece_id=7BZ46KAIEIJDIKHXRLDCMB4Y6E7KSC4QLBOLY2A2NZFA4GVVXD2A, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.525296006 (21h9m13.564869127s) [noise=true , address=45.91.116.6:4975 , piece_id=7U2D53SCSJ5GUGLFWEH6FA66Q5U7EIK7YVBTP5GP5P5OOAVOHQWA, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.525386890 (21h9m13.56496001s) [noise=true , address=45.91.116.10:4083 , piece_id=2XGLF7IS7LC7TU5X2M27UEYQBVX3UML3UEWSFUISJOXF6AVKITTQ, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.525611531 (21h9m13.565184651s) [noise=true , address=45.91.116.7:4009 , piece_id=TYKXIJX7XIMFYKEA5RMORUFVO7UDVKNVBSJAIFUEIB3A76G4HN3Q, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.525648361 (21h9m13.565221481s) [noise=true , address=45.91.116.14:4191 , piece_id=BLZ4EZD5DQ7QQNNRLNCKAB2CQBZGQ2SYKGJJIOZCXNGS4SYOFNXA, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.525870506 (21h9m13.565443626s) [noise=true , address=45.91.116.11:4117 , piece_id=5E2JPHN5T2SETJNOGGAESLRU7PRPUCUDEO25QDW6PA73EKVGMRYQ, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.525935922 (21h9m13.565509042s) [noise=true , address=45.91.116.7:4010 , piece_id=UQRAOTS6AILKBAR6QC3ZD3VATJEYM2T6T4K2KDVU3XRKYQSEB66A, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.526222790 (21h9m13.56579591s) [noise=true , address=45.91.116.13:4153 , piece_id=KUSBCYGRA2XFXZPOYJG4EYGZ7FOSKPKV2EEFQHUND6TPTNQX7AGQ, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.526424746 (21h9m13.565997866s) [noise=true , address=77.90.31.10:4518 , piece_id=XBJTZKIXGIW3WPKU4RWYRZBMAC67YYDYRNAJNBWRYX6SHU3T73WA, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:39.362042598 (21h9m15.401615708s) [upload=18635]: Next returned nil segment Sep 7 10:37:39.362059750 (21h9m15.40163287s) [upload=18635]: Waiting for error group managing segments... Sep 7 10:37:39.669685089 (21h9m15.709258209s) [noise=true , address=45.91.116.14:4191 , piece_id=BLZ4EZD5DQ7QQNNRLNCKAB2CQBZGQ2SYKGJJIOZCXNGS4SYOFNXA, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.679437981 (21h9m15.719011101s) [noise=true , address=77.90.31.10:4519 , piece_id=5RGPSBJO5NDUDD7Y4NH5CYTGF2FB5RO6ADKEP7YUGZFVMCTACVKA, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.679555526 (21h9m15.719128646s) [noise=true , address=45.91.116.12:4225 , piece_id=JQOKRSTWFE56HRHV4BBEL3ZR7T24XBHVYYU4WA4SWJGF432EPOLQ, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.687220604 (21h9m15.726793724s) [noise=true , address=45.91.116.7:4010 , piece_id=UQRAOTS6AILKBAR6QC3ZD3VATJEYM2T6T4K2KDVU3XRKYQSEB66A, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.711092863 (21h9m15.750665983s) [noise=true , address=45.91.116.4:4939 , piece_id=BXLXHF3W5CPAS73RZXC4GL4PPN5PRMVLSSW4ADSW2M2KWRXQY77Q, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.734404882 (21h9m15.773978002s) [noise=true , address=45.91.116.7:4009 , piece_id=TYKXIJX7XIMFYKEA5RMORUFVO7UDVKNVBSJAIFUEIB3A76G4HN3Q, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.743143673 (21h9m15.782716793s) [noise=true , address=45.91.116.11:4119 , piece_id=Y5TR5QSARCJ3HXQZN6OWO7NTLMSGCSLCOEZVJRSB3VZQQL7O57NA, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.748574906 (21h9m15.788148016s) [noise=true , address=45.91.116.6:4974 , piece_id=ZUB4DD6ZXILUSYESNMGNOCKZVTPUY24N52YF2KY3Z36JYNDGRM2A, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.754311293 (21h9m15.793884413s) [noise=true , address=45.91.116.13:4153 , piece_id=KUSBCYGRA2XFXZPOYJG4EYGZ7FOSKPKV2EEFQHUND6TPTNQX7AGQ, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.755384276 (21h9m15.794957396s) [noise=true , address=45.91.116.5:4298 , piece_id=T4KK43URSFHH2LDPBX4CN66G4KVBCWV7JSJM3CADPG6EYGOS2HFQ, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.758010662 (21h9m15.797583782s) [noise=true , address=77.90.31.10:4518 , piece_id=XBJTZKIXGIW3WPKU4RWYRZBMAC67YYDYRNAJNBWRYX6SHU3T73WA, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.774473184 (21h9m15.814046304s) [noise=true , address=45.91.116.6:4973 , piece_id=X63L2GPVONW7ORT5DYOQIHV46EG4DIHEB6TLOZMXKLCQPR5EWDFA, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.817437797 (21h9m15.857010917s) [noise=true , address=45.91.116.11:4117 , piece_id=5E2JPHN5T2SETJNOGGAESLRU7PRPUCUDEO25QDW6PA73EKVGMRYQ, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.817462975 (21h9m15.857036096s) [seg_pos={0 0}, upload=18635]: Segment reached optimal threshold of 13 pieces. Sep 7 10:37:39.817597944 (21h9m15.857171064s) [noise=true , address=45.91.116.6:4975 , piece_id=7U2D53SCSJ5GUGLFWEH6FA66Q5U7EIK7YVBTP5GP5P5OOAVOHQWA, seg_pos={0 0}, upload=18635]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: expected piece hash; context canceled; EOF Sep 7 10:37:39.817683578 (21h9m15.857256698s) [noise=true , address=45.91.116.10:4083 , piece_id=2XGLF7IS7LC7TU5X2M27UEYQBVX3UML3UEWSFUISJOXF6AVKITTQ, seg_pos={0 0}, upload=18635]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: expected piece hash; context canceled; EOF Sep 7 10:37:39.817775434 (21h9m15.857348554s) [noise=true , address=45.91.116.12:4227 , piece_id=7BZ46KAIEIJDIKHXRLDCMB4Y6E7KSC4QLBOLY2A2NZFA4GVVXD2A, seg_pos={0 0}, upload=18635]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH): protocol: expected piece hash; context canceled; EOF Sep 7 10:37:39.817852270 (21h9m15.8574254s) [seg_pos={0 0}, upload=18635]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 7 10:37:39.817955087 (21h9m15.857528217s) [upload=18635]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 7 10:37:39.817961028 (21h9m15.857534148s) [upload=18635]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 7 10:37:40.298833319 (21h9m16.338406429s) [upload=18635]: Done uploading segments... Sep 7 10:37:40.298849160 (21h9m16.33842228s) [upload=18635]: Upload finished. err: <nil> Duration: 7.388238743s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................X...........................] duration: 7.600955882s FAIL (2025-09-07 10:59:06 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Sep 7 10:37:37.259810434 (21h9m13.299383554s) [upload=18635]: Starting upload Sep 7 10:37:37.259869848 (21h9m13.299442968s) [upload=18635]: Done starting upload Sep 7 10:37:37.259991862 (21h9m13.299564982s) [upload=18635]: Uploading segments... Sep 7 10:37:37.260091512 (21h9m13.299664632s) [upload=18635]: Got next segment. Inline: false Sep 7 10:37:37.260124284 (21h9m13.299697405s) [upload=18635]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 7 10:37:37.260129676 (21h9m13.299702796s) [upload=18635]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 7 10:37:37.523765418 (21h9m13.563338538s) [seg_pos={0 0}, upload=18635]: Begin upload segment... Sep 7 10:37:37.523848977 (21h9m13.563422098s) [seg_pos={0 0}, upload=18635]: Done begin upload segment. Sep 7 10:37:37.523875308 (21h9m13.563448418s) [noise=true , address=45.91.116.11:4119 , piece_id=Y5TR5QSARCJ3HXQZN6OWO7NTLMSGCSLCOEZVJRSB3VZQQL7O57NA, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.523927959 (21h9m13.563501079s) [noise=true , address=45.91.116.5:4298 , piece_id=T4KK43URSFHH2LDPBX4CN66G4KVBCWV7JSJM3CADPG6EYGOS2HFQ, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.524262138 (21h9m13.563835258s) [noise=true , address=45.91.116.6:4973 , piece_id=X63L2GPVONW7ORT5DYOQIHV46EG4DIHEB6TLOZMXKLCQPR5EWDFA, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.524469726 (21h9m13.564042846s) [noise=true , address=45.91.116.6:4974 , piece_id=ZUB4DD6ZXILUSYESNMGNOCKZVTPUY24N52YF2KY3Z36JYNDGRM2A, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.524553666 (21h9m13.564126786s) [noise=true , address=45.91.116.4:4939 , piece_id=BXLXHF3W5CPAS73RZXC4GL4PPN5PRMVLSSW4ADSW2M2KWRXQY77Q, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.524760001 (21h9m13.564333121s) [noise=true , address=77.90.31.10:4519 , piece_id=5RGPSBJO5NDUDD7Y4NH5CYTGF2FB5RO6ADKEP7YUGZFVMCTACVKA, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.524839723 (21h9m13.564412843s) [noise=true , address=45.91.116.12:4225 , piece_id=JQOKRSTWFE56HRHV4BBEL3ZR7T24XBHVYYU4WA4SWJGF432EPOLQ, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.525132163 (21h9m13.564705293s) [noise=true , address=45.91.116.12:4227 , piece_id=7BZ46KAIEIJDIKHXRLDCMB4Y6E7KSC4QLBOLY2A2NZFA4GVVXD2A, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.525296006 (21h9m13.564869127s) [noise=true , address=45.91.116.6:4975 , piece_id=7U2D53SCSJ5GUGLFWEH6FA66Q5U7EIK7YVBTP5GP5P5OOAVOHQWA, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.525386890 (21h9m13.56496001s) [noise=true , address=45.91.116.10:4083 , piece_id=2XGLF7IS7LC7TU5X2M27UEYQBVX3UML3UEWSFUISJOXF6AVKITTQ, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.525611531 (21h9m13.565184651s) [noise=true , address=45.91.116.7:4009 , piece_id=TYKXIJX7XIMFYKEA5RMORUFVO7UDVKNVBSJAIFUEIB3A76G4HN3Q, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.525648361 (21h9m13.565221481s) [noise=true , address=45.91.116.14:4191 , piece_id=BLZ4EZD5DQ7QQNNRLNCKAB2CQBZGQ2SYKGJJIOZCXNGS4SYOFNXA, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.525870506 (21h9m13.565443626s) [noise=true , address=45.91.116.11:4117 , piece_id=5E2JPHN5T2SETJNOGGAESLRU7PRPUCUDEO25QDW6PA73EKVGMRYQ, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.525935922 (21h9m13.565509042s) [noise=true , address=45.91.116.7:4010 , piece_id=UQRAOTS6AILKBAR6QC3ZD3VATJEYM2T6T4K2KDVU3XRKYQSEB66A, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.526222790 (21h9m13.56579591s) [noise=true , address=45.91.116.13:4153 , piece_id=KUSBCYGRA2XFXZPOYJG4EYGZ7FOSKPKV2EEFQHUND6TPTNQX7AGQ, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.526424746 (21h9m13.565997866s) [noise=true , address=77.90.31.10:4518 , piece_id=XBJTZKIXGIW3WPKU4RWYRZBMAC67YYDYRNAJNBWRYX6SHU3T73WA, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:39.362042598 (21h9m15.401615708s) [upload=18635]: Next returned nil segment Sep 7 10:37:39.362059750 (21h9m15.40163287s) [upload=18635]: Waiting for error group managing segments... Sep 7 10:37:39.669685089 (21h9m15.709258209s) [noise=true , address=45.91.116.14:4191 , piece_id=BLZ4EZD5DQ7QQNNRLNCKAB2CQBZGQ2SYKGJJIOZCXNGS4SYOFNXA, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.679437981 (21h9m15.719011101s) [noise=true , address=77.90.31.10:4519 , piece_id=5RGPSBJO5NDUDD7Y4NH5CYTGF2FB5RO6ADKEP7YUGZFVMCTACVKA, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.679555526 (21h9m15.719128646s) [noise=true , address=45.91.116.12:4225 , piece_id=JQOKRSTWFE56HRHV4BBEL3ZR7T24XBHVYYU4WA4SWJGF432EPOLQ, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.687220604 (21h9m15.726793724s) [noise=true , address=45.91.116.7:4010 , piece_id=UQRAOTS6AILKBAR6QC3ZD3VATJEYM2T6T4K2KDVU3XRKYQSEB66A, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.711092863 (21h9m15.750665983s) [noise=true , address=45.91.116.4:4939 , piece_id=BXLXHF3W5CPAS73RZXC4GL4PPN5PRMVLSSW4ADSW2M2KWRXQY77Q, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.734404882 (21h9m15.773978002s) [noise=true , address=45.91.116.7:4009 , piece_id=TYKXIJX7XIMFYKEA5RMORUFVO7UDVKNVBSJAIFUEIB3A76G4HN3Q, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.743143673 (21h9m15.782716793s) [noise=true , address=45.91.116.11:4119 , piece_id=Y5TR5QSARCJ3HXQZN6OWO7NTLMSGCSLCOEZVJRSB3VZQQL7O57NA, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.748574906 (21h9m15.788148016s) [noise=true , address=45.91.116.6:4974 , piece_id=ZUB4DD6ZXILUSYESNMGNOCKZVTPUY24N52YF2KY3Z36JYNDGRM2A, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.754311293 (21h9m15.793884413s) [noise=true , address=45.91.116.13:4153 , piece_id=KUSBCYGRA2XFXZPOYJG4EYGZ7FOSKPKV2EEFQHUND6TPTNQX7AGQ, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.755384276 (21h9m15.794957396s) [noise=true , address=45.91.116.5:4298 , piece_id=T4KK43URSFHH2LDPBX4CN66G4KVBCWV7JSJM3CADPG6EYGOS2HFQ, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.758010662 (21h9m15.797583782s) [noise=true , address=77.90.31.10:4518 , piece_id=XBJTZKIXGIW3WPKU4RWYRZBMAC67YYDYRNAJNBWRYX6SHU3T73WA, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.774473184 (21h9m15.814046304s) [noise=true , address=45.91.116.6:4973 , piece_id=X63L2GPVONW7ORT5DYOQIHV46EG4DIHEB6TLOZMXKLCQPR5EWDFA, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.817437797 (21h9m15.857010917s) [noise=true , address=45.91.116.11:4117 , piece_id=5E2JPHN5T2SETJNOGGAESLRU7PRPUCUDEO25QDW6PA73EKVGMRYQ, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.817462975 (21h9m15.857036096s) [seg_pos={0 0}, upload=18635]: Segment reached optimal threshold of 13 pieces. Sep 7 10:37:39.817597944 (21h9m15.857171064s) [noise=true , address=45.91.116.6:4975 , piece_id=7U2D53SCSJ5GUGLFWEH6FA66Q5U7EIK7YVBTP5GP5P5OOAVOHQWA, seg_pos={0 0}, upload=18635]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: expected piece hash; context canceled; EOF Sep 7 10:37:39.817683578 (21h9m15.857256698s) [noise=true , address=45.91.116.10:4083 , piece_id=2XGLF7IS7LC7TU5X2M27UEYQBVX3UML3UEWSFUISJOXF6AVKITTQ, seg_pos={0 0}, upload=18635]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: expected piece hash; context canceled; EOF Sep 7 10:37:39.817775434 (21h9m15.857348554s) [noise=true , address=45.91.116.12:4227 , piece_id=7BZ46KAIEIJDIKHXRLDCMB4Y6E7KSC4QLBOLY2A2NZFA4GVVXD2A, seg_pos={0 0}, upload=18635]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH): protocol: expected piece hash; context canceled; EOF Sep 7 10:37:39.817852270 (21h9m15.8574254s) [seg_pos={0 0}, upload=18635]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 7 10:37:39.817955087 (21h9m15.857528217s) [upload=18635]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 7 10:37:39.817961028 (21h9m15.857534148s) [upload=18635]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 7 10:37:40.298833319 (21h9m16.338406429s) [upload=18635]: Done uploading segments... Sep 7 10:37:40.298849160 (21h9m16.33842228s) [upload=18635]: Upload finished. err: <nil> Duration: 7.600972985s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................X..........................] duration: 9.937514519s FAIL (2025-09-07 10:58:37 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Sep 7 10:37:37.259810434 (21h9m13.299383554s) [upload=18635]: Starting upload Sep 7 10:37:37.259869848 (21h9m13.299442968s) [upload=18635]: Done starting upload Sep 7 10:37:37.259991862 (21h9m13.299564982s) [upload=18635]: Uploading segments... Sep 7 10:37:37.260091512 (21h9m13.299664632s) [upload=18635]: Got next segment. Inline: false Sep 7 10:37:37.260124284 (21h9m13.299697405s) [upload=18635]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 7 10:37:37.260129676 (21h9m13.299702796s) [upload=18635]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 7 10:37:37.523765418 (21h9m13.563338538s) [seg_pos={0 0}, upload=18635]: Begin upload segment... Sep 7 10:37:37.523848977 (21h9m13.563422098s) [seg_pos={0 0}, upload=18635]: Done begin upload segment. Sep 7 10:37:37.523875308 (21h9m13.563448418s) [noise=true , address=45.91.116.11:4119 , piece_id=Y5TR5QSARCJ3HXQZN6OWO7NTLMSGCSLCOEZVJRSB3VZQQL7O57NA, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.523927959 (21h9m13.563501079s) [noise=true , address=45.91.116.5:4298 , piece_id=T4KK43URSFHH2LDPBX4CN66G4KVBCWV7JSJM3CADPG6EYGOS2HFQ, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.524262138 (21h9m13.563835258s) [noise=true , address=45.91.116.6:4973 , piece_id=X63L2GPVONW7ORT5DYOQIHV46EG4DIHEB6TLOZMXKLCQPR5EWDFA, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.524469726 (21h9m13.564042846s) [noise=true , address=45.91.116.6:4974 , piece_id=ZUB4DD6ZXILUSYESNMGNOCKZVTPUY24N52YF2KY3Z36JYNDGRM2A, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.524553666 (21h9m13.564126786s) [noise=true , address=45.91.116.4:4939 , piece_id=BXLXHF3W5CPAS73RZXC4GL4PPN5PRMVLSSW4ADSW2M2KWRXQY77Q, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.524760001 (21h9m13.564333121s) [noise=true , address=77.90.31.10:4519 , piece_id=5RGPSBJO5NDUDD7Y4NH5CYTGF2FB5RO6ADKEP7YUGZFVMCTACVKA, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.524839723 (21h9m13.564412843s) [noise=true , address=45.91.116.12:4225 , piece_id=JQOKRSTWFE56HRHV4BBEL3ZR7T24XBHVYYU4WA4SWJGF432EPOLQ, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.525132163 (21h9m13.564705293s) [noise=true , address=45.91.116.12:4227 , piece_id=7BZ46KAIEIJDIKHXRLDCMB4Y6E7KSC4QLBOLY2A2NZFA4GVVXD2A, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.525296006 (21h9m13.564869127s) [noise=true , address=45.91.116.6:4975 , piece_id=7U2D53SCSJ5GUGLFWEH6FA66Q5U7EIK7YVBTP5GP5P5OOAVOHQWA, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.525386890 (21h9m13.56496001s) [noise=true , address=45.91.116.10:4083 , piece_id=2XGLF7IS7LC7TU5X2M27UEYQBVX3UML3UEWSFUISJOXF6AVKITTQ, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.525611531 (21h9m13.565184651s) [noise=true , address=45.91.116.7:4009 , piece_id=TYKXIJX7XIMFYKEA5RMORUFVO7UDVKNVBSJAIFUEIB3A76G4HN3Q, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.525648361 (21h9m13.565221481s) [noise=true , address=45.91.116.14:4191 , piece_id=BLZ4EZD5DQ7QQNNRLNCKAB2CQBZGQ2SYKGJJIOZCXNGS4SYOFNXA, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.525870506 (21h9m13.565443626s) [noise=true , address=45.91.116.11:4117 , piece_id=5E2JPHN5T2SETJNOGGAESLRU7PRPUCUDEO25QDW6PA73EKVGMRYQ, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.525935922 (21h9m13.565509042s) [noise=true , address=45.91.116.7:4010 , piece_id=UQRAOTS6AILKBAR6QC3ZD3VATJEYM2T6T4K2KDVU3XRKYQSEB66A, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.526222790 (21h9m13.56579591s) [noise=true , address=45.91.116.13:4153 , piece_id=KUSBCYGRA2XFXZPOYJG4EYGZ7FOSKPKV2EEFQHUND6TPTNQX7AGQ, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.526424746 (21h9m13.565997866s) [noise=true , address=77.90.31.10:4518 , piece_id=XBJTZKIXGIW3WPKU4RWYRZBMAC67YYDYRNAJNBWRYX6SHU3T73WA, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:39.362042598 (21h9m15.401615708s) [upload=18635]: Next returned nil segment Sep 7 10:37:39.362059750 (21h9m15.40163287s) [upload=18635]: Waiting for error group managing segments... Sep 7 10:37:39.669685089 (21h9m15.709258209s) [noise=true , address=45.91.116.14:4191 , piece_id=BLZ4EZD5DQ7QQNNRLNCKAB2CQBZGQ2SYKGJJIOZCXNGS4SYOFNXA, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.679437981 (21h9m15.719011101s) [noise=true , address=77.90.31.10:4519 , piece_id=5RGPSBJO5NDUDD7Y4NH5CYTGF2FB5RO6ADKEP7YUGZFVMCTACVKA, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.679555526 (21h9m15.719128646s) [noise=true , address=45.91.116.12:4225 , piece_id=JQOKRSTWFE56HRHV4BBEL3ZR7T24XBHVYYU4WA4SWJGF432EPOLQ, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.687220604 (21h9m15.726793724s) [noise=true , address=45.91.116.7:4010 , piece_id=UQRAOTS6AILKBAR6QC3ZD3VATJEYM2T6T4K2KDVU3XRKYQSEB66A, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.711092863 (21h9m15.750665983s) [noise=true , address=45.91.116.4:4939 , piece_id=BXLXHF3W5CPAS73RZXC4GL4PPN5PRMVLSSW4ADSW2M2KWRXQY77Q, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.734404882 (21h9m15.773978002s) [noise=true , address=45.91.116.7:4009 , piece_id=TYKXIJX7XIMFYKEA5RMORUFVO7UDVKNVBSJAIFUEIB3A76G4HN3Q, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.743143673 (21h9m15.782716793s) [noise=true , address=45.91.116.11:4119 , piece_id=Y5TR5QSARCJ3HXQZN6OWO7NTLMSGCSLCOEZVJRSB3VZQQL7O57NA, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.748574906 (21h9m15.788148016s) [noise=true , address=45.91.116.6:4974 , piece_id=ZUB4DD6ZXILUSYESNMGNOCKZVTPUY24N52YF2KY3Z36JYNDGRM2A, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.754311293 (21h9m15.793884413s) [noise=true , address=45.91.116.13:4153 , piece_id=KUSBCYGRA2XFXZPOYJG4EYGZ7FOSKPKV2EEFQHUND6TPTNQX7AGQ, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.755384276 (21h9m15.794957396s) [noise=true , address=45.91.116.5:4298 , piece_id=T4KK43URSFHH2LDPBX4CN66G4KVBCWV7JSJM3CADPG6EYGOS2HFQ, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.758010662 (21h9m15.797583782s) [noise=true , address=77.90.31.10:4518 , piece_id=XBJTZKIXGIW3WPKU4RWYRZBMAC67YYDYRNAJNBWRYX6SHU3T73WA, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.774473184 (21h9m15.814046304s) [noise=true , address=45.91.116.6:4973 , piece_id=X63L2GPVONW7ORT5DYOQIHV46EG4DIHEB6TLOZMXKLCQPR5EWDFA, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.817437797 (21h9m15.857010917s) [noise=true , address=45.91.116.11:4117 , piece_id=5E2JPHN5T2SETJNOGGAESLRU7PRPUCUDEO25QDW6PA73EKVGMRYQ, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.817462975 (21h9m15.857036096s) [seg_pos={0 0}, upload=18635]: Segment reached optimal threshold of 13 pieces. Sep 7 10:37:39.817597944 (21h9m15.857171064s) [noise=true , address=45.91.116.6:4975 , piece_id=7U2D53SCSJ5GUGLFWEH6FA66Q5U7EIK7YVBTP5GP5P5OOAVOHQWA, seg_pos={0 0}, upload=18635]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: expected piece hash; context canceled; EOF Sep 7 10:37:39.817683578 (21h9m15.857256698s) [noise=true , address=45.91.116.10:4083 , piece_id=2XGLF7IS7LC7TU5X2M27UEYQBVX3UML3UEWSFUISJOXF6AVKITTQ, seg_pos={0 0}, upload=18635]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: expected piece hash; context canceled; EOF Sep 7 10:37:39.817775434 (21h9m15.857348554s) [noise=true , address=45.91.116.12:4227 , piece_id=7BZ46KAIEIJDIKHXRLDCMB4Y6E7KSC4QLBOLY2A2NZFA4GVVXD2A, seg_pos={0 0}, upload=18635]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH): protocol: expected piece hash; context canceled; EOF Sep 7 10:37:39.817852270 (21h9m15.8574254s) [seg_pos={0 0}, upload=18635]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 7 10:37:39.817955087 (21h9m15.857528217s) [upload=18635]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 7 10:37:39.817961028 (21h9m15.857534148s) [upload=18635]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 7 10:37:40.298833319 (21h9m16.338406429s) [upload=18635]: Done uploading segments... Sep 7 10:37:40.298849160 (21h9m16.33842228s) [upload=18635]: Upload finished. err: <nil> Duration: 9.937548685s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................X.........................] duration: 11.139745732s FAIL (2025-09-07 10:57:29 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Sep 7 10:37:37.259810434 (21h9m13.299383554s) [upload=18635]: Starting upload Sep 7 10:37:37.259869848 (21h9m13.299442968s) [upload=18635]: Done starting upload Sep 7 10:37:37.259991862 (21h9m13.299564982s) [upload=18635]: Uploading segments... Sep 7 10:37:37.260091512 (21h9m13.299664632s) [upload=18635]: Got next segment. Inline: false Sep 7 10:37:37.260124284 (21h9m13.299697405s) [upload=18635]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 7 10:37:37.260129676 (21h9m13.299702796s) [upload=18635]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 7 10:37:37.523765418 (21h9m13.563338538s) [seg_pos={0 0}, upload=18635]: Begin upload segment... Sep 7 10:37:37.523848977 (21h9m13.563422098s) [seg_pos={0 0}, upload=18635]: Done begin upload segment. Sep 7 10:37:37.523875308 (21h9m13.563448418s) [noise=true , address=45.91.116.11:4119 , piece_id=Y5TR5QSARCJ3HXQZN6OWO7NTLMSGCSLCOEZVJRSB3VZQQL7O57NA, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.523927959 (21h9m13.563501079s) [noise=true , address=45.91.116.5:4298 , piece_id=T4KK43URSFHH2LDPBX4CN66G4KVBCWV7JSJM3CADPG6EYGOS2HFQ, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.524262138 (21h9m13.563835258s) [noise=true , address=45.91.116.6:4973 , piece_id=X63L2GPVONW7ORT5DYOQIHV46EG4DIHEB6TLOZMXKLCQPR5EWDFA, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.524469726 (21h9m13.564042846s) [noise=true , address=45.91.116.6:4974 , piece_id=ZUB4DD6ZXILUSYESNMGNOCKZVTPUY24N52YF2KY3Z36JYNDGRM2A, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.524553666 (21h9m13.564126786s) [noise=true , address=45.91.116.4:4939 , piece_id=BXLXHF3W5CPAS73RZXC4GL4PPN5PRMVLSSW4ADSW2M2KWRXQY77Q, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.524760001 (21h9m13.564333121s) [noise=true , address=77.90.31.10:4519 , piece_id=5RGPSBJO5NDUDD7Y4NH5CYTGF2FB5RO6ADKEP7YUGZFVMCTACVKA, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.524839723 (21h9m13.564412843s) [noise=true , address=45.91.116.12:4225 , piece_id=JQOKRSTWFE56HRHV4BBEL3ZR7T24XBHVYYU4WA4SWJGF432EPOLQ, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.525132163 (21h9m13.564705293s) [noise=true , address=45.91.116.12:4227 , piece_id=7BZ46KAIEIJDIKHXRLDCMB4Y6E7KSC4QLBOLY2A2NZFA4GVVXD2A, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.525296006 (21h9m13.564869127s) [noise=true , address=45.91.116.6:4975 , piece_id=7U2D53SCSJ5GUGLFWEH6FA66Q5U7EIK7YVBTP5GP5P5OOAVOHQWA, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.525386890 (21h9m13.56496001s) [noise=true , address=45.91.116.10:4083 , piece_id=2XGLF7IS7LC7TU5X2M27UEYQBVX3UML3UEWSFUISJOXF6AVKITTQ, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.525611531 (21h9m13.565184651s) [noise=true , address=45.91.116.7:4009 , piece_id=TYKXIJX7XIMFYKEA5RMORUFVO7UDVKNVBSJAIFUEIB3A76G4HN3Q, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.525648361 (21h9m13.565221481s) [noise=true , address=45.91.116.14:4191 , piece_id=BLZ4EZD5DQ7QQNNRLNCKAB2CQBZGQ2SYKGJJIOZCXNGS4SYOFNXA, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.525870506 (21h9m13.565443626s) [noise=true , address=45.91.116.11:4117 , piece_id=5E2JPHN5T2SETJNOGGAESLRU7PRPUCUDEO25QDW6PA73EKVGMRYQ, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.525935922 (21h9m13.565509042s) [noise=true , address=45.91.116.7:4010 , piece_id=UQRAOTS6AILKBAR6QC3ZD3VATJEYM2T6T4K2KDVU3XRKYQSEB66A, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.526222790 (21h9m13.56579591s) [noise=true , address=45.91.116.13:4153 , piece_id=KUSBCYGRA2XFXZPOYJG4EYGZ7FOSKPKV2EEFQHUND6TPTNQX7AGQ, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.526424746 (21h9m13.565997866s) [noise=true , address=77.90.31.10:4518 , piece_id=XBJTZKIXGIW3WPKU4RWYRZBMAC67YYDYRNAJNBWRYX6SHU3T73WA, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:39.362042598 (21h9m15.401615708s) [upload=18635]: Next returned nil segment Sep 7 10:37:39.362059750 (21h9m15.40163287s) [upload=18635]: Waiting for error group managing segments... Sep 7 10:37:39.669685089 (21h9m15.709258209s) [noise=true , address=45.91.116.14:4191 , piece_id=BLZ4EZD5DQ7QQNNRLNCKAB2CQBZGQ2SYKGJJIOZCXNGS4SYOFNXA, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.679437981 (21h9m15.719011101s) [noise=true , address=77.90.31.10:4519 , piece_id=5RGPSBJO5NDUDD7Y4NH5CYTGF2FB5RO6ADKEP7YUGZFVMCTACVKA, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.679555526 (21h9m15.719128646s) [noise=true , address=45.91.116.12:4225 , piece_id=JQOKRSTWFE56HRHV4BBEL3ZR7T24XBHVYYU4WA4SWJGF432EPOLQ, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.687220604 (21h9m15.726793724s) [noise=true , address=45.91.116.7:4010 , piece_id=UQRAOTS6AILKBAR6QC3ZD3VATJEYM2T6T4K2KDVU3XRKYQSEB66A, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.711092863 (21h9m15.750665983s) [noise=true , address=45.91.116.4:4939 , piece_id=BXLXHF3W5CPAS73RZXC4GL4PPN5PRMVLSSW4ADSW2M2KWRXQY77Q, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.734404882 (21h9m15.773978002s) [noise=true , address=45.91.116.7:4009 , piece_id=TYKXIJX7XIMFYKEA5RMORUFVO7UDVKNVBSJAIFUEIB3A76G4HN3Q, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.743143673 (21h9m15.782716793s) [noise=true , address=45.91.116.11:4119 , piece_id=Y5TR5QSARCJ3HXQZN6OWO7NTLMSGCSLCOEZVJRSB3VZQQL7O57NA, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.748574906 (21h9m15.788148016s) [noise=true , address=45.91.116.6:4974 , piece_id=ZUB4DD6ZXILUSYESNMGNOCKZVTPUY24N52YF2KY3Z36JYNDGRM2A, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.754311293 (21h9m15.793884413s) [noise=true , address=45.91.116.13:4153 , piece_id=KUSBCYGRA2XFXZPOYJG4EYGZ7FOSKPKV2EEFQHUND6TPTNQX7AGQ, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.755384276 (21h9m15.794957396s) [noise=true , address=45.91.116.5:4298 , piece_id=T4KK43URSFHH2LDPBX4CN66G4KVBCWV7JSJM3CADPG6EYGOS2HFQ, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.758010662 (21h9m15.797583782s) [noise=true , address=77.90.31.10:4518 , piece_id=XBJTZKIXGIW3WPKU4RWYRZBMAC67YYDYRNAJNBWRYX6SHU3T73WA, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.774473184 (21h9m15.814046304s) [noise=true , address=45.91.116.6:4973 , piece_id=X63L2GPVONW7ORT5DYOQIHV46EG4DIHEB6TLOZMXKLCQPR5EWDFA, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.817437797 (21h9m15.857010917s) [noise=true , address=45.91.116.11:4117 , piece_id=5E2JPHN5T2SETJNOGGAESLRU7PRPUCUDEO25QDW6PA73EKVGMRYQ, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.817462975 (21h9m15.857036096s) [seg_pos={0 0}, upload=18635]: Segment reached optimal threshold of 13 pieces. Sep 7 10:37:39.817597944 (21h9m15.857171064s) [noise=true , address=45.91.116.6:4975 , piece_id=7U2D53SCSJ5GUGLFWEH6FA66Q5U7EIK7YVBTP5GP5P5OOAVOHQWA, seg_pos={0 0}, upload=18635]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: expected piece hash; context canceled; EOF Sep 7 10:37:39.817683578 (21h9m15.857256698s) [noise=true , address=45.91.116.10:4083 , piece_id=2XGLF7IS7LC7TU5X2M27UEYQBVX3UML3UEWSFUISJOXF6AVKITTQ, seg_pos={0 0}, upload=18635]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: expected piece hash; context canceled; EOF Sep 7 10:37:39.817775434 (21h9m15.857348554s) [noise=true , address=45.91.116.12:4227 , piece_id=7BZ46KAIEIJDIKHXRLDCMB4Y6E7KSC4QLBOLY2A2NZFA4GVVXD2A, seg_pos={0 0}, upload=18635]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH): protocol: expected piece hash; context canceled; EOF Sep 7 10:37:39.817852270 (21h9m15.8574254s) [seg_pos={0 0}, upload=18635]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 7 10:37:39.817955087 (21h9m15.857528217s) [upload=18635]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 7 10:37:39.817961028 (21h9m15.857534148s) [upload=18635]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 7 10:37:40.298833319 (21h9m16.338406429s) [upload=18635]: Done uploading segments... Sep 7 10:37:40.298849160 (21h9m16.33842228s) [upload=18635]: Upload finished. err: <nil> Duration: 11.139766391s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................X........................] duration: 15.476877338s FAIL (2025-09-07 10:56:46 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Sep 7 10:37:37.259810434 (21h9m13.299383554s) [upload=18635]: Starting upload Sep 7 10:37:37.259869848 (21h9m13.299442968s) [upload=18635]: Done starting upload Sep 7 10:37:37.259991862 (21h9m13.299564982s) [upload=18635]: Uploading segments... Sep 7 10:37:37.260091512 (21h9m13.299664632s) [upload=18635]: Got next segment. Inline: false Sep 7 10:37:37.260124284 (21h9m13.299697405s) [upload=18635]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 7 10:37:37.260129676 (21h9m13.299702796s) [upload=18635]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 7 10:37:37.523765418 (21h9m13.563338538s) [seg_pos={0 0}, upload=18635]: Begin upload segment... Sep 7 10:37:37.523848977 (21h9m13.563422098s) [seg_pos={0 0}, upload=18635]: Done begin upload segment. Sep 7 10:37:37.523875308 (21h9m13.563448418s) [noise=true , address=45.91.116.11:4119 , piece_id=Y5TR5QSARCJ3HXQZN6OWO7NTLMSGCSLCOEZVJRSB3VZQQL7O57NA, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.523927959 (21h9m13.563501079s) [noise=true , address=45.91.116.5:4298 , piece_id=T4KK43URSFHH2LDPBX4CN66G4KVBCWV7JSJM3CADPG6EYGOS2HFQ, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.524262138 (21h9m13.563835258s) [noise=true , address=45.91.116.6:4973 , piece_id=X63L2GPVONW7ORT5DYOQIHV46EG4DIHEB6TLOZMXKLCQPR5EWDFA, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.524469726 (21h9m13.564042846s) [noise=true , address=45.91.116.6:4974 , piece_id=ZUB4DD6ZXILUSYESNMGNOCKZVTPUY24N52YF2KY3Z36JYNDGRM2A, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.524553666 (21h9m13.564126786s) [noise=true , address=45.91.116.4:4939 , piece_id=BXLXHF3W5CPAS73RZXC4GL4PPN5PRMVLSSW4ADSW2M2KWRXQY77Q, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.524760001 (21h9m13.564333121s) [noise=true , address=77.90.31.10:4519 , piece_id=5RGPSBJO5NDUDD7Y4NH5CYTGF2FB5RO6ADKEP7YUGZFVMCTACVKA, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.524839723 (21h9m13.564412843s) [noise=true , address=45.91.116.12:4225 , piece_id=JQOKRSTWFE56HRHV4BBEL3ZR7T24XBHVYYU4WA4SWJGF432EPOLQ, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.525132163 (21h9m13.564705293s) [noise=true , address=45.91.116.12:4227 , piece_id=7BZ46KAIEIJDIKHXRLDCMB4Y6E7KSC4QLBOLY2A2NZFA4GVVXD2A, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.525296006 (21h9m13.564869127s) [noise=true , address=45.91.116.6:4975 , piece_id=7U2D53SCSJ5GUGLFWEH6FA66Q5U7EIK7YVBTP5GP5P5OOAVOHQWA, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.525386890 (21h9m13.56496001s) [noise=true , address=45.91.116.10:4083 , piece_id=2XGLF7IS7LC7TU5X2M27UEYQBVX3UML3UEWSFUISJOXF6AVKITTQ, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.525611531 (21h9m13.565184651s) [noise=true , address=45.91.116.7:4009 , piece_id=TYKXIJX7XIMFYKEA5RMORUFVO7UDVKNVBSJAIFUEIB3A76G4HN3Q, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.525648361 (21h9m13.565221481s) [noise=true , address=45.91.116.14:4191 , piece_id=BLZ4EZD5DQ7QQNNRLNCKAB2CQBZGQ2SYKGJJIOZCXNGS4SYOFNXA, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.525870506 (21h9m13.565443626s) [noise=true , address=45.91.116.11:4117 , piece_id=5E2JPHN5T2SETJNOGGAESLRU7PRPUCUDEO25QDW6PA73EKVGMRYQ, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.525935922 (21h9m13.565509042s) [noise=true , address=45.91.116.7:4010 , piece_id=UQRAOTS6AILKBAR6QC3ZD3VATJEYM2T6T4K2KDVU3XRKYQSEB66A, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.526222790 (21h9m13.56579591s) [noise=true , address=45.91.116.13:4153 , piece_id=KUSBCYGRA2XFXZPOYJG4EYGZ7FOSKPKV2EEFQHUND6TPTNQX7AGQ, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.526424746 (21h9m13.565997866s) [noise=true , address=77.90.31.10:4518 , piece_id=XBJTZKIXGIW3WPKU4RWYRZBMAC67YYDYRNAJNBWRYX6SHU3T73WA, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:39.362042598 (21h9m15.401615708s) [upload=18635]: Next returned nil segment Sep 7 10:37:39.362059750 (21h9m15.40163287s) [upload=18635]: Waiting for error group managing segments... Sep 7 10:37:39.669685089 (21h9m15.709258209s) [noise=true , address=45.91.116.14:4191 , piece_id=BLZ4EZD5DQ7QQNNRLNCKAB2CQBZGQ2SYKGJJIOZCXNGS4SYOFNXA, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.679437981 (21h9m15.719011101s) [noise=true , address=77.90.31.10:4519 , piece_id=5RGPSBJO5NDUDD7Y4NH5CYTGF2FB5RO6ADKEP7YUGZFVMCTACVKA, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.679555526 (21h9m15.719128646s) [noise=true , address=45.91.116.12:4225 , piece_id=JQOKRSTWFE56HRHV4BBEL3ZR7T24XBHVYYU4WA4SWJGF432EPOLQ, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.687220604 (21h9m15.726793724s) [noise=true , address=45.91.116.7:4010 , piece_id=UQRAOTS6AILKBAR6QC3ZD3VATJEYM2T6T4K2KDVU3XRKYQSEB66A, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.711092863 (21h9m15.750665983s) [noise=true , address=45.91.116.4:4939 , piece_id=BXLXHF3W5CPAS73RZXC4GL4PPN5PRMVLSSW4ADSW2M2KWRXQY77Q, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.734404882 (21h9m15.773978002s) [noise=true , address=45.91.116.7:4009 , piece_id=TYKXIJX7XIMFYKEA5RMORUFVO7UDVKNVBSJAIFUEIB3A76G4HN3Q, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.743143673 (21h9m15.782716793s) [noise=true , address=45.91.116.11:4119 , piece_id=Y5TR5QSARCJ3HXQZN6OWO7NTLMSGCSLCOEZVJRSB3VZQQL7O57NA, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.748574906 (21h9m15.788148016s) [noise=true , address=45.91.116.6:4974 , piece_id=ZUB4DD6ZXILUSYESNMGNOCKZVTPUY24N52YF2KY3Z36JYNDGRM2A, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.754311293 (21h9m15.793884413s) [noise=true , address=45.91.116.13:4153 , piece_id=KUSBCYGRA2XFXZPOYJG4EYGZ7FOSKPKV2EEFQHUND6TPTNQX7AGQ, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.755384276 (21h9m15.794957396s) [noise=true , address=45.91.116.5:4298 , piece_id=T4KK43URSFHH2LDPBX4CN66G4KVBCWV7JSJM3CADPG6EYGOS2HFQ, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.758010662 (21h9m15.797583782s) [noise=true , address=77.90.31.10:4518 , piece_id=XBJTZKIXGIW3WPKU4RWYRZBMAC67YYDYRNAJNBWRYX6SHU3T73WA, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.774473184 (21h9m15.814046304s) [noise=true , address=45.91.116.6:4973 , piece_id=X63L2GPVONW7ORT5DYOQIHV46EG4DIHEB6TLOZMXKLCQPR5EWDFA, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.817437797 (21h9m15.857010917s) [noise=true , address=45.91.116.11:4117 , piece_id=5E2JPHN5T2SETJNOGGAESLRU7PRPUCUDEO25QDW6PA73EKVGMRYQ, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.817462975 (21h9m15.857036096s) [seg_pos={0 0}, upload=18635]: Segment reached optimal threshold of 13 pieces. Sep 7 10:37:39.817597944 (21h9m15.857171064s) [noise=true , address=45.91.116.6:4975 , piece_id=7U2D53SCSJ5GUGLFWEH6FA66Q5U7EIK7YVBTP5GP5P5OOAVOHQWA, seg_pos={0 0}, upload=18635]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: expected piece hash; context canceled; EOF Sep 7 10:37:39.817683578 (21h9m15.857256698s) [noise=true , address=45.91.116.10:4083 , piece_id=2XGLF7IS7LC7TU5X2M27UEYQBVX3UML3UEWSFUISJOXF6AVKITTQ, seg_pos={0 0}, upload=18635]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: expected piece hash; context canceled; EOF Sep 7 10:37:39.817775434 (21h9m15.857348554s) [noise=true , address=45.91.116.12:4227 , piece_id=7BZ46KAIEIJDIKHXRLDCMB4Y6E7KSC4QLBOLY2A2NZFA4GVVXD2A, seg_pos={0 0}, upload=18635]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH): protocol: expected piece hash; context canceled; EOF Sep 7 10:37:39.817852270 (21h9m15.8574254s) [seg_pos={0 0}, upload=18635]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 7 10:37:39.817955087 (21h9m15.857528217s) [upload=18635]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 7 10:37:39.817961028 (21h9m15.857534148s) [upload=18635]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 7 10:37:40.298833319 (21h9m16.338406429s) [upload=18635]: Done uploading segments... Sep 7 10:37:40.298849160 (21h9m16.33842228s) [upload=18635]: Upload finished. err: <nil> Duration: 15.476907326s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................X.......................] duration: 7.385229943s FAIL (2025-09-07 10:56:08 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Sep 7 10:37:37.259810434 (21h9m13.299383554s) [upload=18635]: Starting upload Sep 7 10:37:37.259869848 (21h9m13.299442968s) [upload=18635]: Done starting upload Sep 7 10:37:37.259991862 (21h9m13.299564982s) [upload=18635]: Uploading segments... Sep 7 10:37:37.260091512 (21h9m13.299664632s) [upload=18635]: Got next segment. Inline: false Sep 7 10:37:37.260124284 (21h9m13.299697405s) [upload=18635]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 7 10:37:37.260129676 (21h9m13.299702796s) [upload=18635]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 7 10:37:37.523765418 (21h9m13.563338538s) [seg_pos={0 0}, upload=18635]: Begin upload segment... Sep 7 10:37:37.523848977 (21h9m13.563422098s) [seg_pos={0 0}, upload=18635]: Done begin upload segment. Sep 7 10:37:37.523875308 (21h9m13.563448418s) [noise=true , address=45.91.116.11:4119 , piece_id=Y5TR5QSARCJ3HXQZN6OWO7NTLMSGCSLCOEZVJRSB3VZQQL7O57NA, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.523927959 (21h9m13.563501079s) [noise=true , address=45.91.116.5:4298 , piece_id=T4KK43URSFHH2LDPBX4CN66G4KVBCWV7JSJM3CADPG6EYGOS2HFQ, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.524262138 (21h9m13.563835258s) [noise=true , address=45.91.116.6:4973 , piece_id=X63L2GPVONW7ORT5DYOQIHV46EG4DIHEB6TLOZMXKLCQPR5EWDFA, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.524469726 (21h9m13.564042846s) [noise=true , address=45.91.116.6:4974 , piece_id=ZUB4DD6ZXILUSYESNMGNOCKZVTPUY24N52YF2KY3Z36JYNDGRM2A, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.524553666 (21h9m13.564126786s) [noise=true , address=45.91.116.4:4939 , piece_id=BXLXHF3W5CPAS73RZXC4GL4PPN5PRMVLSSW4ADSW2M2KWRXQY77Q, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.524760001 (21h9m13.564333121s) [noise=true , address=77.90.31.10:4519 , piece_id=5RGPSBJO5NDUDD7Y4NH5CYTGF2FB5RO6ADKEP7YUGZFVMCTACVKA, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.524839723 (21h9m13.564412843s) [noise=true , address=45.91.116.12:4225 , piece_id=JQOKRSTWFE56HRHV4BBEL3ZR7T24XBHVYYU4WA4SWJGF432EPOLQ, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.525132163 (21h9m13.564705293s) [noise=true , address=45.91.116.12:4227 , piece_id=7BZ46KAIEIJDIKHXRLDCMB4Y6E7KSC4QLBOLY2A2NZFA4GVVXD2A, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.525296006 (21h9m13.564869127s) [noise=true , address=45.91.116.6:4975 , piece_id=7U2D53SCSJ5GUGLFWEH6FA66Q5U7EIK7YVBTP5GP5P5OOAVOHQWA, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.525386890 (21h9m13.56496001s) [noise=true , address=45.91.116.10:4083 , piece_id=2XGLF7IS7LC7TU5X2M27UEYQBVX3UML3UEWSFUISJOXF6AVKITTQ, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.525611531 (21h9m13.565184651s) [noise=true , address=45.91.116.7:4009 , piece_id=TYKXIJX7XIMFYKEA5RMORUFVO7UDVKNVBSJAIFUEIB3A76G4HN3Q, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.525648361 (21h9m13.565221481s) [noise=true , address=45.91.116.14:4191 , piece_id=BLZ4EZD5DQ7QQNNRLNCKAB2CQBZGQ2SYKGJJIOZCXNGS4SYOFNXA, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.525870506 (21h9m13.565443626s) [noise=true , address=45.91.116.11:4117 , piece_id=5E2JPHN5T2SETJNOGGAESLRU7PRPUCUDEO25QDW6PA73EKVGMRYQ, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.525935922 (21h9m13.565509042s) [noise=true , address=45.91.116.7:4010 , piece_id=UQRAOTS6AILKBAR6QC3ZD3VATJEYM2T6T4K2KDVU3XRKYQSEB66A, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.526222790 (21h9m13.56579591s) [noise=true , address=45.91.116.13:4153 , piece_id=KUSBCYGRA2XFXZPOYJG4EYGZ7FOSKPKV2EEFQHUND6TPTNQX7AGQ, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.526424746 (21h9m13.565997866s) [noise=true , address=77.90.31.10:4518 , piece_id=XBJTZKIXGIW3WPKU4RWYRZBMAC67YYDYRNAJNBWRYX6SHU3T73WA, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:39.362042598 (21h9m15.401615708s) [upload=18635]: Next returned nil segment Sep 7 10:37:39.362059750 (21h9m15.40163287s) [upload=18635]: Waiting for error group managing segments... Sep 7 10:37:39.669685089 (21h9m15.709258209s) [noise=true , address=45.91.116.14:4191 , piece_id=BLZ4EZD5DQ7QQNNRLNCKAB2CQBZGQ2SYKGJJIOZCXNGS4SYOFNXA, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.679437981 (21h9m15.719011101s) [noise=true , address=77.90.31.10:4519 , piece_id=5RGPSBJO5NDUDD7Y4NH5CYTGF2FB5RO6ADKEP7YUGZFVMCTACVKA, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.679555526 (21h9m15.719128646s) [noise=true , address=45.91.116.12:4225 , piece_id=JQOKRSTWFE56HRHV4BBEL3ZR7T24XBHVYYU4WA4SWJGF432EPOLQ, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.687220604 (21h9m15.726793724s) [noise=true , address=45.91.116.7:4010 , piece_id=UQRAOTS6AILKBAR6QC3ZD3VATJEYM2T6T4K2KDVU3XRKYQSEB66A, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.711092863 (21h9m15.750665983s) [noise=true , address=45.91.116.4:4939 , piece_id=BXLXHF3W5CPAS73RZXC4GL4PPN5PRMVLSSW4ADSW2M2KWRXQY77Q, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.734404882 (21h9m15.773978002s) [noise=true , address=45.91.116.7:4009 , piece_id=TYKXIJX7XIMFYKEA5RMORUFVO7UDVKNVBSJAIFUEIB3A76G4HN3Q, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.743143673 (21h9m15.782716793s) [noise=true , address=45.91.116.11:4119 , piece_id=Y5TR5QSARCJ3HXQZN6OWO7NTLMSGCSLCOEZVJRSB3VZQQL7O57NA, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.748574906 (21h9m15.788148016s) [noise=true , address=45.91.116.6:4974 , piece_id=ZUB4DD6ZXILUSYESNMGNOCKZVTPUY24N52YF2KY3Z36JYNDGRM2A, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.754311293 (21h9m15.793884413s) [noise=true , address=45.91.116.13:4153 , piece_id=KUSBCYGRA2XFXZPOYJG4EYGZ7FOSKPKV2EEFQHUND6TPTNQX7AGQ, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.755384276 (21h9m15.794957396s) [noise=true , address=45.91.116.5:4298 , piece_id=T4KK43URSFHH2LDPBX4CN66G4KVBCWV7JSJM3CADPG6EYGOS2HFQ, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.758010662 (21h9m15.797583782s) [noise=true , address=77.90.31.10:4518 , piece_id=XBJTZKIXGIW3WPKU4RWYRZBMAC67YYDYRNAJNBWRYX6SHU3T73WA, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.774473184 (21h9m15.814046304s) [noise=true , address=45.91.116.6:4973 , piece_id=X63L2GPVONW7ORT5DYOQIHV46EG4DIHEB6TLOZMXKLCQPR5EWDFA, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.817437797 (21h9m15.857010917s) [noise=true , address=45.91.116.11:4117 , piece_id=5E2JPHN5T2SETJNOGGAESLRU7PRPUCUDEO25QDW6PA73EKVGMRYQ, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.817462975 (21h9m15.857036096s) [seg_pos={0 0}, upload=18635]: Segment reached optimal threshold of 13 pieces. Sep 7 10:37:39.817597944 (21h9m15.857171064s) [noise=true , address=45.91.116.6:4975 , piece_id=7U2D53SCSJ5GUGLFWEH6FA66Q5U7EIK7YVBTP5GP5P5OOAVOHQWA, seg_pos={0 0}, upload=18635]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: expected piece hash; context canceled; EOF Sep 7 10:37:39.817683578 (21h9m15.857256698s) [noise=true , address=45.91.116.10:4083 , piece_id=2XGLF7IS7LC7TU5X2M27UEYQBVX3UML3UEWSFUISJOXF6AVKITTQ, seg_pos={0 0}, upload=18635]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: expected piece hash; context canceled; EOF Sep 7 10:37:39.817775434 (21h9m15.857348554s) [noise=true , address=45.91.116.12:4227 , piece_id=7BZ46KAIEIJDIKHXRLDCMB4Y6E7KSC4QLBOLY2A2NZFA4GVVXD2A, seg_pos={0 0}, upload=18635]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH): protocol: expected piece hash; context canceled; EOF Sep 7 10:37:39.817852270 (21h9m15.8574254s) [seg_pos={0 0}, upload=18635]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 7 10:37:39.817955087 (21h9m15.857528217s) [upload=18635]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 7 10:37:39.817961028 (21h9m15.857534148s) [upload=18635]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 7 10:37:40.298833319 (21h9m16.338406429s) [upload=18635]: Done uploading segments... Sep 7 10:37:40.298849160 (21h9m16.33842228s) [upload=18635]: Upload finished. err: <nil> Duration: 7.385265893s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................X......................] duration: 11.144931475s FAIL (2025-09-07 10:55:00 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Sep 7 10:37:37.259810434 (21h9m13.299383554s) [upload=18635]: Starting upload Sep 7 10:37:37.259869848 (21h9m13.299442968s) [upload=18635]: Done starting upload Sep 7 10:37:37.259991862 (21h9m13.299564982s) [upload=18635]: Uploading segments... Sep 7 10:37:37.260091512 (21h9m13.299664632s) [upload=18635]: Got next segment. Inline: false Sep 7 10:37:37.260124284 (21h9m13.299697405s) [upload=18635]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 7 10:37:37.260129676 (21h9m13.299702796s) [upload=18635]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 7 10:37:37.523765418 (21h9m13.563338538s) [seg_pos={0 0}, upload=18635]: Begin upload segment... Sep 7 10:37:37.523848977 (21h9m13.563422098s) [seg_pos={0 0}, upload=18635]: Done begin upload segment. Sep 7 10:37:37.523875308 (21h9m13.563448418s) [noise=true , address=45.91.116.11:4119 , piece_id=Y5TR5QSARCJ3HXQZN6OWO7NTLMSGCSLCOEZVJRSB3VZQQL7O57NA, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.523927959 (21h9m13.563501079s) [noise=true , address=45.91.116.5:4298 , piece_id=T4KK43URSFHH2LDPBX4CN66G4KVBCWV7JSJM3CADPG6EYGOS2HFQ, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.524262138 (21h9m13.563835258s) [noise=true , address=45.91.116.6:4973 , piece_id=X63L2GPVONW7ORT5DYOQIHV46EG4DIHEB6TLOZMXKLCQPR5EWDFA, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.524469726 (21h9m13.564042846s) [noise=true , address=45.91.116.6:4974 , piece_id=ZUB4DD6ZXILUSYESNMGNOCKZVTPUY24N52YF2KY3Z36JYNDGRM2A, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.524553666 (21h9m13.564126786s) [noise=true , address=45.91.116.4:4939 , piece_id=BXLXHF3W5CPAS73RZXC4GL4PPN5PRMVLSSW4ADSW2M2KWRXQY77Q, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.524760001 (21h9m13.564333121s) [noise=true , address=77.90.31.10:4519 , piece_id=5RGPSBJO5NDUDD7Y4NH5CYTGF2FB5RO6ADKEP7YUGZFVMCTACVKA, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.524839723 (21h9m13.564412843s) [noise=true , address=45.91.116.12:4225 , piece_id=JQOKRSTWFE56HRHV4BBEL3ZR7T24XBHVYYU4WA4SWJGF432EPOLQ, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.525132163 (21h9m13.564705293s) [noise=true , address=45.91.116.12:4227 , piece_id=7BZ46KAIEIJDIKHXRLDCMB4Y6E7KSC4QLBOLY2A2NZFA4GVVXD2A, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.525296006 (21h9m13.564869127s) [noise=true , address=45.91.116.6:4975 , piece_id=7U2D53SCSJ5GUGLFWEH6FA66Q5U7EIK7YVBTP5GP5P5OOAVOHQWA, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.525386890 (21h9m13.56496001s) [noise=true , address=45.91.116.10:4083 , piece_id=2XGLF7IS7LC7TU5X2M27UEYQBVX3UML3UEWSFUISJOXF6AVKITTQ, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.525611531 (21h9m13.565184651s) [noise=true , address=45.91.116.7:4009 , piece_id=TYKXIJX7XIMFYKEA5RMORUFVO7UDVKNVBSJAIFUEIB3A76G4HN3Q, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.525648361 (21h9m13.565221481s) [noise=true , address=45.91.116.14:4191 , piece_id=BLZ4EZD5DQ7QQNNRLNCKAB2CQBZGQ2SYKGJJIOZCXNGS4SYOFNXA, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.525870506 (21h9m13.565443626s) [noise=true , address=45.91.116.11:4117 , piece_id=5E2JPHN5T2SETJNOGGAESLRU7PRPUCUDEO25QDW6PA73EKVGMRYQ, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.525935922 (21h9m13.565509042s) [noise=true , address=45.91.116.7:4010 , piece_id=UQRAOTS6AILKBAR6QC3ZD3VATJEYM2T6T4K2KDVU3XRKYQSEB66A, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.526222790 (21h9m13.56579591s) [noise=true , address=45.91.116.13:4153 , piece_id=KUSBCYGRA2XFXZPOYJG4EYGZ7FOSKPKV2EEFQHUND6TPTNQX7AGQ, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.526424746 (21h9m13.565997866s) [noise=true , address=77.90.31.10:4518 , piece_id=XBJTZKIXGIW3WPKU4RWYRZBMAC67YYDYRNAJNBWRYX6SHU3T73WA, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:39.362042598 (21h9m15.401615708s) [upload=18635]: Next returned nil segment Sep 7 10:37:39.362059750 (21h9m15.40163287s) [upload=18635]: Waiting for error group managing segments... Sep 7 10:37:39.669685089 (21h9m15.709258209s) [noise=true , address=45.91.116.14:4191 , piece_id=BLZ4EZD5DQ7QQNNRLNCKAB2CQBZGQ2SYKGJJIOZCXNGS4SYOFNXA, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.679437981 (21h9m15.719011101s) [noise=true , address=77.90.31.10:4519 , piece_id=5RGPSBJO5NDUDD7Y4NH5CYTGF2FB5RO6ADKEP7YUGZFVMCTACVKA, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.679555526 (21h9m15.719128646s) [noise=true , address=45.91.116.12:4225 , piece_id=JQOKRSTWFE56HRHV4BBEL3ZR7T24XBHVYYU4WA4SWJGF432EPOLQ, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.687220604 (21h9m15.726793724s) [noise=true , address=45.91.116.7:4010 , piece_id=UQRAOTS6AILKBAR6QC3ZD3VATJEYM2T6T4K2KDVU3XRKYQSEB66A, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.711092863 (21h9m15.750665983s) [noise=true , address=45.91.116.4:4939 , piece_id=BXLXHF3W5CPAS73RZXC4GL4PPN5PRMVLSSW4ADSW2M2KWRXQY77Q, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.734404882 (21h9m15.773978002s) [noise=true , address=45.91.116.7:4009 , piece_id=TYKXIJX7XIMFYKEA5RMORUFVO7UDVKNVBSJAIFUEIB3A76G4HN3Q, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.743143673 (21h9m15.782716793s) [noise=true , address=45.91.116.11:4119 , piece_id=Y5TR5QSARCJ3HXQZN6OWO7NTLMSGCSLCOEZVJRSB3VZQQL7O57NA, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.748574906 (21h9m15.788148016s) [noise=true , address=45.91.116.6:4974 , piece_id=ZUB4DD6ZXILUSYESNMGNOCKZVTPUY24N52YF2KY3Z36JYNDGRM2A, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.754311293 (21h9m15.793884413s) [noise=true , address=45.91.116.13:4153 , piece_id=KUSBCYGRA2XFXZPOYJG4EYGZ7FOSKPKV2EEFQHUND6TPTNQX7AGQ, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.755384276 (21h9m15.794957396s) [noise=true , address=45.91.116.5:4298 , piece_id=T4KK43URSFHH2LDPBX4CN66G4KVBCWV7JSJM3CADPG6EYGOS2HFQ, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.758010662 (21h9m15.797583782s) [noise=true , address=77.90.31.10:4518 , piece_id=XBJTZKIXGIW3WPKU4RWYRZBMAC67YYDYRNAJNBWRYX6SHU3T73WA, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.774473184 (21h9m15.814046304s) [noise=true , address=45.91.116.6:4973 , piece_id=X63L2GPVONW7ORT5DYOQIHV46EG4DIHEB6TLOZMXKLCQPR5EWDFA, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.817437797 (21h9m15.857010917s) [noise=true , address=45.91.116.11:4117 , piece_id=5E2JPHN5T2SETJNOGGAESLRU7PRPUCUDEO25QDW6PA73EKVGMRYQ, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.817462975 (21h9m15.857036096s) [seg_pos={0 0}, upload=18635]: Segment reached optimal threshold of 13 pieces. Sep 7 10:37:39.817597944 (21h9m15.857171064s) [noise=true , address=45.91.116.6:4975 , piece_id=7U2D53SCSJ5GUGLFWEH6FA66Q5U7EIK7YVBTP5GP5P5OOAVOHQWA, seg_pos={0 0}, upload=18635]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: expected piece hash; context canceled; EOF Sep 7 10:37:39.817683578 (21h9m15.857256698s) [noise=true , address=45.91.116.10:4083 , piece_id=2XGLF7IS7LC7TU5X2M27UEYQBVX3UML3UEWSFUISJOXF6AVKITTQ, seg_pos={0 0}, upload=18635]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: expected piece hash; context canceled; EOF Sep 7 10:37:39.817775434 (21h9m15.857348554s) [noise=true , address=45.91.116.12:4227 , piece_id=7BZ46KAIEIJDIKHXRLDCMB4Y6E7KSC4QLBOLY2A2NZFA4GVVXD2A, seg_pos={0 0}, upload=18635]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH): protocol: expected piece hash; context canceled; EOF Sep 7 10:37:39.817852270 (21h9m15.8574254s) [seg_pos={0 0}, upload=18635]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 7 10:37:39.817955087 (21h9m15.857528217s) [upload=18635]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 7 10:37:39.817961028 (21h9m15.857534148s) [upload=18635]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 7 10:37:40.298833319 (21h9m16.338406429s) [upload=18635]: Done uploading segments... Sep 7 10:37:40.298849160 (21h9m16.33842228s) [upload=18635]: Upload finished. err: <nil> Duration: 11.144957936s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................X.....................] duration: 12.75714603s FAIL (2025-09-07 10:54:09 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Sep 7 10:37:37.259810434 (21h9m13.299383554s) [upload=18635]: Starting upload Sep 7 10:37:37.259869848 (21h9m13.299442968s) [upload=18635]: Done starting upload Sep 7 10:37:37.259991862 (21h9m13.299564982s) [upload=18635]: Uploading segments... Sep 7 10:37:37.260091512 (21h9m13.299664632s) [upload=18635]: Got next segment. Inline: false Sep 7 10:37:37.260124284 (21h9m13.299697405s) [upload=18635]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 7 10:37:37.260129676 (21h9m13.299702796s) [upload=18635]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 7 10:37:37.523765418 (21h9m13.563338538s) [seg_pos={0 0}, upload=18635]: Begin upload segment... Sep 7 10:37:37.523848977 (21h9m13.563422098s) [seg_pos={0 0}, upload=18635]: Done begin upload segment. Sep 7 10:37:37.523875308 (21h9m13.563448418s) [noise=true , address=45.91.116.11:4119 , piece_id=Y5TR5QSARCJ3HXQZN6OWO7NTLMSGCSLCOEZVJRSB3VZQQL7O57NA, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.523927959 (21h9m13.563501079s) [noise=true , address=45.91.116.5:4298 , piece_id=T4KK43URSFHH2LDPBX4CN66G4KVBCWV7JSJM3CADPG6EYGOS2HFQ, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.524262138 (21h9m13.563835258s) [noise=true , address=45.91.116.6:4973 , piece_id=X63L2GPVONW7ORT5DYOQIHV46EG4DIHEB6TLOZMXKLCQPR5EWDFA, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.524469726 (21h9m13.564042846s) [noise=true , address=45.91.116.6:4974 , piece_id=ZUB4DD6ZXILUSYESNMGNOCKZVTPUY24N52YF2KY3Z36JYNDGRM2A, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.524553666 (21h9m13.564126786s) [noise=true , address=45.91.116.4:4939 , piece_id=BXLXHF3W5CPAS73RZXC4GL4PPN5PRMVLSSW4ADSW2M2KWRXQY77Q, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.524760001 (21h9m13.564333121s) [noise=true , address=77.90.31.10:4519 , piece_id=5RGPSBJO5NDUDD7Y4NH5CYTGF2FB5RO6ADKEP7YUGZFVMCTACVKA, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.524839723 (21h9m13.564412843s) [noise=true , address=45.91.116.12:4225 , piece_id=JQOKRSTWFE56HRHV4BBEL3ZR7T24XBHVYYU4WA4SWJGF432EPOLQ, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.525132163 (21h9m13.564705293s) [noise=true , address=45.91.116.12:4227 , piece_id=7BZ46KAIEIJDIKHXRLDCMB4Y6E7KSC4QLBOLY2A2NZFA4GVVXD2A, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.525296006 (21h9m13.564869127s) [noise=true , address=45.91.116.6:4975 , piece_id=7U2D53SCSJ5GUGLFWEH6FA66Q5U7EIK7YVBTP5GP5P5OOAVOHQWA, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.525386890 (21h9m13.56496001s) [noise=true , address=45.91.116.10:4083 , piece_id=2XGLF7IS7LC7TU5X2M27UEYQBVX3UML3UEWSFUISJOXF6AVKITTQ, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.525611531 (21h9m13.565184651s) [noise=true , address=45.91.116.7:4009 , piece_id=TYKXIJX7XIMFYKEA5RMORUFVO7UDVKNVBSJAIFUEIB3A76G4HN3Q, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.525648361 (21h9m13.565221481s) [noise=true , address=45.91.116.14:4191 , piece_id=BLZ4EZD5DQ7QQNNRLNCKAB2CQBZGQ2SYKGJJIOZCXNGS4SYOFNXA, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.525870506 (21h9m13.565443626s) [noise=true , address=45.91.116.11:4117 , piece_id=5E2JPHN5T2SETJNOGGAESLRU7PRPUCUDEO25QDW6PA73EKVGMRYQ, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.525935922 (21h9m13.565509042s) [noise=true , address=45.91.116.7:4010 , piece_id=UQRAOTS6AILKBAR6QC3ZD3VATJEYM2T6T4K2KDVU3XRKYQSEB66A, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.526222790 (21h9m13.56579591s) [noise=true , address=45.91.116.13:4153 , piece_id=KUSBCYGRA2XFXZPOYJG4EYGZ7FOSKPKV2EEFQHUND6TPTNQX7AGQ, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.526424746 (21h9m13.565997866s) [noise=true , address=77.90.31.10:4518 , piece_id=XBJTZKIXGIW3WPKU4RWYRZBMAC67YYDYRNAJNBWRYX6SHU3T73WA, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:39.362042598 (21h9m15.401615708s) [upload=18635]: Next returned nil segment Sep 7 10:37:39.362059750 (21h9m15.40163287s) [upload=18635]: Waiting for error group managing segments... Sep 7 10:37:39.669685089 (21h9m15.709258209s) [noise=true , address=45.91.116.14:4191 , piece_id=BLZ4EZD5DQ7QQNNRLNCKAB2CQBZGQ2SYKGJJIOZCXNGS4SYOFNXA, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.679437981 (21h9m15.719011101s) [noise=true , address=77.90.31.10:4519 , piece_id=5RGPSBJO5NDUDD7Y4NH5CYTGF2FB5RO6ADKEP7YUGZFVMCTACVKA, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.679555526 (21h9m15.719128646s) [noise=true , address=45.91.116.12:4225 , piece_id=JQOKRSTWFE56HRHV4BBEL3ZR7T24XBHVYYU4WA4SWJGF432EPOLQ, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.687220604 (21h9m15.726793724s) [noise=true , address=45.91.116.7:4010 , piece_id=UQRAOTS6AILKBAR6QC3ZD3VATJEYM2T6T4K2KDVU3XRKYQSEB66A, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.711092863 (21h9m15.750665983s) [noise=true , address=45.91.116.4:4939 , piece_id=BXLXHF3W5CPAS73RZXC4GL4PPN5PRMVLSSW4ADSW2M2KWRXQY77Q, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.734404882 (21h9m15.773978002s) [noise=true , address=45.91.116.7:4009 , piece_id=TYKXIJX7XIMFYKEA5RMORUFVO7UDVKNVBSJAIFUEIB3A76G4HN3Q, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.743143673 (21h9m15.782716793s) [noise=true , address=45.91.116.11:4119 , piece_id=Y5TR5QSARCJ3HXQZN6OWO7NTLMSGCSLCOEZVJRSB3VZQQL7O57NA, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.748574906 (21h9m15.788148016s) [noise=true , address=45.91.116.6:4974 , piece_id=ZUB4DD6ZXILUSYESNMGNOCKZVTPUY24N52YF2KY3Z36JYNDGRM2A, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.754311293 (21h9m15.793884413s) [noise=true , address=45.91.116.13:4153 , piece_id=KUSBCYGRA2XFXZPOYJG4EYGZ7FOSKPKV2EEFQHUND6TPTNQX7AGQ, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.755384276 (21h9m15.794957396s) [noise=true , address=45.91.116.5:4298 , piece_id=T4KK43URSFHH2LDPBX4CN66G4KVBCWV7JSJM3CADPG6EYGOS2HFQ, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.758010662 (21h9m15.797583782s) [noise=true , address=77.90.31.10:4518 , piece_id=XBJTZKIXGIW3WPKU4RWYRZBMAC67YYDYRNAJNBWRYX6SHU3T73WA, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.774473184 (21h9m15.814046304s) [noise=true , address=45.91.116.6:4973 , piece_id=X63L2GPVONW7ORT5DYOQIHV46EG4DIHEB6TLOZMXKLCQPR5EWDFA, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.817437797 (21h9m15.857010917s) [noise=true , address=45.91.116.11:4117 , piece_id=5E2JPHN5T2SETJNOGGAESLRU7PRPUCUDEO25QDW6PA73EKVGMRYQ, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.817462975 (21h9m15.857036096s) [seg_pos={0 0}, upload=18635]: Segment reached optimal threshold of 13 pieces. Sep 7 10:37:39.817597944 (21h9m15.857171064s) [noise=true , address=45.91.116.6:4975 , piece_id=7U2D53SCSJ5GUGLFWEH6FA66Q5U7EIK7YVBTP5GP5P5OOAVOHQWA, seg_pos={0 0}, upload=18635]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: expected piece hash; context canceled; EOF Sep 7 10:37:39.817683578 (21h9m15.857256698s) [noise=true , address=45.91.116.10:4083 , piece_id=2XGLF7IS7LC7TU5X2M27UEYQBVX3UML3UEWSFUISJOXF6AVKITTQ, seg_pos={0 0}, upload=18635]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: expected piece hash; context canceled; EOF Sep 7 10:37:39.817775434 (21h9m15.857348554s) [noise=true , address=45.91.116.12:4227 , piece_id=7BZ46KAIEIJDIKHXRLDCMB4Y6E7KSC4QLBOLY2A2NZFA4GVVXD2A, seg_pos={0 0}, upload=18635]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH): protocol: expected piece hash; context canceled; EOF Sep 7 10:37:39.817852270 (21h9m15.8574254s) [seg_pos={0 0}, upload=18635]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 7 10:37:39.817955087 (21h9m15.857528217s) [upload=18635]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 7 10:37:39.817961028 (21h9m15.857534148s) [upload=18635]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 7 10:37:40.298833319 (21h9m16.338406429s) [upload=18635]: Done uploading segments... Sep 7 10:37:40.298849160 (21h9m16.33842228s) [upload=18635]: Upload finished. err: <nil> Duration: 12.757170727s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................X....................] duration: 6.821923624s FAIL (2025-09-07 10:53:29 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Sep 7 10:37:37.259810434 (21h9m13.299383554s) [upload=18635]: Starting upload Sep 7 10:37:37.259869848 (21h9m13.299442968s) [upload=18635]: Done starting upload Sep 7 10:37:37.259991862 (21h9m13.299564982s) [upload=18635]: Uploading segments... Sep 7 10:37:37.260091512 (21h9m13.299664632s) [upload=18635]: Got next segment. Inline: false Sep 7 10:37:37.260124284 (21h9m13.299697405s) [upload=18635]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 7 10:37:37.260129676 (21h9m13.299702796s) [upload=18635]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 7 10:37:37.523765418 (21h9m13.563338538s) [seg_pos={0 0}, upload=18635]: Begin upload segment... Sep 7 10:37:37.523848977 (21h9m13.563422098s) [seg_pos={0 0}, upload=18635]: Done begin upload segment. Sep 7 10:37:37.523875308 (21h9m13.563448418s) [noise=true , address=45.91.116.11:4119 , piece_id=Y5TR5QSARCJ3HXQZN6OWO7NTLMSGCSLCOEZVJRSB3VZQQL7O57NA, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.523927959 (21h9m13.563501079s) [noise=true , address=45.91.116.5:4298 , piece_id=T4KK43URSFHH2LDPBX4CN66G4KVBCWV7JSJM3CADPG6EYGOS2HFQ, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.524262138 (21h9m13.563835258s) [noise=true , address=45.91.116.6:4973 , piece_id=X63L2GPVONW7ORT5DYOQIHV46EG4DIHEB6TLOZMXKLCQPR5EWDFA, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.524469726 (21h9m13.564042846s) [noise=true , address=45.91.116.6:4974 , piece_id=ZUB4DD6ZXILUSYESNMGNOCKZVTPUY24N52YF2KY3Z36JYNDGRM2A, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.524553666 (21h9m13.564126786s) [noise=true , address=45.91.116.4:4939 , piece_id=BXLXHF3W5CPAS73RZXC4GL4PPN5PRMVLSSW4ADSW2M2KWRXQY77Q, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.524760001 (21h9m13.564333121s) [noise=true , address=77.90.31.10:4519 , piece_id=5RGPSBJO5NDUDD7Y4NH5CYTGF2FB5RO6ADKEP7YUGZFVMCTACVKA, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.524839723 (21h9m13.564412843s) [noise=true , address=45.91.116.12:4225 , piece_id=JQOKRSTWFE56HRHV4BBEL3ZR7T24XBHVYYU4WA4SWJGF432EPOLQ, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.525132163 (21h9m13.564705293s) [noise=true , address=45.91.116.12:4227 , piece_id=7BZ46KAIEIJDIKHXRLDCMB4Y6E7KSC4QLBOLY2A2NZFA4GVVXD2A, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.525296006 (21h9m13.564869127s) [noise=true , address=45.91.116.6:4975 , piece_id=7U2D53SCSJ5GUGLFWEH6FA66Q5U7EIK7YVBTP5GP5P5OOAVOHQWA, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.525386890 (21h9m13.56496001s) [noise=true , address=45.91.116.10:4083 , piece_id=2XGLF7IS7LC7TU5X2M27UEYQBVX3UML3UEWSFUISJOXF6AVKITTQ, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.525611531 (21h9m13.565184651s) [noise=true , address=45.91.116.7:4009 , piece_id=TYKXIJX7XIMFYKEA5RMORUFVO7UDVKNVBSJAIFUEIB3A76G4HN3Q, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.525648361 (21h9m13.565221481s) [noise=true , address=45.91.116.14:4191 , piece_id=BLZ4EZD5DQ7QQNNRLNCKAB2CQBZGQ2SYKGJJIOZCXNGS4SYOFNXA, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.525870506 (21h9m13.565443626s) [noise=true , address=45.91.116.11:4117 , piece_id=5E2JPHN5T2SETJNOGGAESLRU7PRPUCUDEO25QDW6PA73EKVGMRYQ, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.525935922 (21h9m13.565509042s) [noise=true , address=45.91.116.7:4010 , piece_id=UQRAOTS6AILKBAR6QC3ZD3VATJEYM2T6T4K2KDVU3XRKYQSEB66A, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.526222790 (21h9m13.56579591s) [noise=true , address=45.91.116.13:4153 , piece_id=KUSBCYGRA2XFXZPOYJG4EYGZ7FOSKPKV2EEFQHUND6TPTNQX7AGQ, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.526424746 (21h9m13.565997866s) [noise=true , address=77.90.31.10:4518 , piece_id=XBJTZKIXGIW3WPKU4RWYRZBMAC67YYDYRNAJNBWRYX6SHU3T73WA, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:39.362042598 (21h9m15.401615708s) [upload=18635]: Next returned nil segment Sep 7 10:37:39.362059750 (21h9m15.40163287s) [upload=18635]: Waiting for error group managing segments... Sep 7 10:37:39.669685089 (21h9m15.709258209s) [noise=true , address=45.91.116.14:4191 , piece_id=BLZ4EZD5DQ7QQNNRLNCKAB2CQBZGQ2SYKGJJIOZCXNGS4SYOFNXA, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.679437981 (21h9m15.719011101s) [noise=true , address=77.90.31.10:4519 , piece_id=5RGPSBJO5NDUDD7Y4NH5CYTGF2FB5RO6ADKEP7YUGZFVMCTACVKA, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.679555526 (21h9m15.719128646s) [noise=true , address=45.91.116.12:4225 , piece_id=JQOKRSTWFE56HRHV4BBEL3ZR7T24XBHVYYU4WA4SWJGF432EPOLQ, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.687220604 (21h9m15.726793724s) [noise=true , address=45.91.116.7:4010 , piece_id=UQRAOTS6AILKBAR6QC3ZD3VATJEYM2T6T4K2KDVU3XRKYQSEB66A, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.711092863 (21h9m15.750665983s) [noise=true , address=45.91.116.4:4939 , piece_id=BXLXHF3W5CPAS73RZXC4GL4PPN5PRMVLSSW4ADSW2M2KWRXQY77Q, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.734404882 (21h9m15.773978002s) [noise=true , address=45.91.116.7:4009 , piece_id=TYKXIJX7XIMFYKEA5RMORUFVO7UDVKNVBSJAIFUEIB3A76G4HN3Q, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.743143673 (21h9m15.782716793s) [noise=true , address=45.91.116.11:4119 , piece_id=Y5TR5QSARCJ3HXQZN6OWO7NTLMSGCSLCOEZVJRSB3VZQQL7O57NA, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.748574906 (21h9m15.788148016s) [noise=true , address=45.91.116.6:4974 , piece_id=ZUB4DD6ZXILUSYESNMGNOCKZVTPUY24N52YF2KY3Z36JYNDGRM2A, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.754311293 (21h9m15.793884413s) [noise=true , address=45.91.116.13:4153 , piece_id=KUSBCYGRA2XFXZPOYJG4EYGZ7FOSKPKV2EEFQHUND6TPTNQX7AGQ, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.755384276 (21h9m15.794957396s) [noise=true , address=45.91.116.5:4298 , piece_id=T4KK43URSFHH2LDPBX4CN66G4KVBCWV7JSJM3CADPG6EYGOS2HFQ, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.758010662 (21h9m15.797583782s) [noise=true , address=77.90.31.10:4518 , piece_id=XBJTZKIXGIW3WPKU4RWYRZBMAC67YYDYRNAJNBWRYX6SHU3T73WA, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.774473184 (21h9m15.814046304s) [noise=true , address=45.91.116.6:4973 , piece_id=X63L2GPVONW7ORT5DYOQIHV46EG4DIHEB6TLOZMXKLCQPR5EWDFA, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.817437797 (21h9m15.857010917s) [noise=true , address=45.91.116.11:4117 , piece_id=5E2JPHN5T2SETJNOGGAESLRU7PRPUCUDEO25QDW6PA73EKVGMRYQ, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.817462975 (21h9m15.857036096s) [seg_pos={0 0}, upload=18635]: Segment reached optimal threshold of 13 pieces. Sep 7 10:37:39.817597944 (21h9m15.857171064s) [noise=true , address=45.91.116.6:4975 , piece_id=7U2D53SCSJ5GUGLFWEH6FA66Q5U7EIK7YVBTP5GP5P5OOAVOHQWA, seg_pos={0 0}, upload=18635]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: expected piece hash; context canceled; EOF Sep 7 10:37:39.817683578 (21h9m15.857256698s) [noise=true , address=45.91.116.10:4083 , piece_id=2XGLF7IS7LC7TU5X2M27UEYQBVX3UML3UEWSFUISJOXF6AVKITTQ, seg_pos={0 0}, upload=18635]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: expected piece hash; context canceled; EOF Sep 7 10:37:39.817775434 (21h9m15.857348554s) [noise=true , address=45.91.116.12:4227 , piece_id=7BZ46KAIEIJDIKHXRLDCMB4Y6E7KSC4QLBOLY2A2NZFA4GVVXD2A, seg_pos={0 0}, upload=18635]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH): protocol: expected piece hash; context canceled; EOF Sep 7 10:37:39.817852270 (21h9m15.8574254s) [seg_pos={0 0}, upload=18635]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 7 10:37:39.817955087 (21h9m15.857528217s) [upload=18635]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 7 10:37:39.817961028 (21h9m15.857534148s) [upload=18635]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 7 10:37:40.298833319 (21h9m16.338406429s) [upload=18635]: Done uploading segments... Sep 7 10:37:40.298849160 (21h9m16.33842228s) [upload=18635]: Upload finished. err: <nil> Duration: 6.821952138s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................X...................] duration: 7.434143112s FAIL (2025-09-07 10:53:10 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Sep 7 10:37:37.259810434 (21h9m13.299383554s) [upload=18635]: Starting upload Sep 7 10:37:37.259869848 (21h9m13.299442968s) [upload=18635]: Done starting upload Sep 7 10:37:37.259991862 (21h9m13.299564982s) [upload=18635]: Uploading segments... Sep 7 10:37:37.260091512 (21h9m13.299664632s) [upload=18635]: Got next segment. Inline: false Sep 7 10:37:37.260124284 (21h9m13.299697405s) [upload=18635]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 7 10:37:37.260129676 (21h9m13.299702796s) [upload=18635]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 7 10:37:37.523765418 (21h9m13.563338538s) [seg_pos={0 0}, upload=18635]: Begin upload segment... Sep 7 10:37:37.523848977 (21h9m13.563422098s) [seg_pos={0 0}, upload=18635]: Done begin upload segment. Sep 7 10:37:37.523875308 (21h9m13.563448418s) [noise=true , address=45.91.116.11:4119 , piece_id=Y5TR5QSARCJ3HXQZN6OWO7NTLMSGCSLCOEZVJRSB3VZQQL7O57NA, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.523927959 (21h9m13.563501079s) [noise=true , address=45.91.116.5:4298 , piece_id=T4KK43URSFHH2LDPBX4CN66G4KVBCWV7JSJM3CADPG6EYGOS2HFQ, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.524262138 (21h9m13.563835258s) [noise=true , address=45.91.116.6:4973 , piece_id=X63L2GPVONW7ORT5DYOQIHV46EG4DIHEB6TLOZMXKLCQPR5EWDFA, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.524469726 (21h9m13.564042846s) [noise=true , address=45.91.116.6:4974 , piece_id=ZUB4DD6ZXILUSYESNMGNOCKZVTPUY24N52YF2KY3Z36JYNDGRM2A, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.524553666 (21h9m13.564126786s) [noise=true , address=45.91.116.4:4939 , piece_id=BXLXHF3W5CPAS73RZXC4GL4PPN5PRMVLSSW4ADSW2M2KWRXQY77Q, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.524760001 (21h9m13.564333121s) [noise=true , address=77.90.31.10:4519 , piece_id=5RGPSBJO5NDUDD7Y4NH5CYTGF2FB5RO6ADKEP7YUGZFVMCTACVKA, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.524839723 (21h9m13.564412843s) [noise=true , address=45.91.116.12:4225 , piece_id=JQOKRSTWFE56HRHV4BBEL3ZR7T24XBHVYYU4WA4SWJGF432EPOLQ, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.525132163 (21h9m13.564705293s) [noise=true , address=45.91.116.12:4227 , piece_id=7BZ46KAIEIJDIKHXRLDCMB4Y6E7KSC4QLBOLY2A2NZFA4GVVXD2A, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.525296006 (21h9m13.564869127s) [noise=true , address=45.91.116.6:4975 , piece_id=7U2D53SCSJ5GUGLFWEH6FA66Q5U7EIK7YVBTP5GP5P5OOAVOHQWA, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.525386890 (21h9m13.56496001s) [noise=true , address=45.91.116.10:4083 , piece_id=2XGLF7IS7LC7TU5X2M27UEYQBVX3UML3UEWSFUISJOXF6AVKITTQ, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.525611531 (21h9m13.565184651s) [noise=true , address=45.91.116.7:4009 , piece_id=TYKXIJX7XIMFYKEA5RMORUFVO7UDVKNVBSJAIFUEIB3A76G4HN3Q, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.525648361 (21h9m13.565221481s) [noise=true , address=45.91.116.14:4191 , piece_id=BLZ4EZD5DQ7QQNNRLNCKAB2CQBZGQ2SYKGJJIOZCXNGS4SYOFNXA, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.525870506 (21h9m13.565443626s) [noise=true , address=45.91.116.11:4117 , piece_id=5E2JPHN5T2SETJNOGGAESLRU7PRPUCUDEO25QDW6PA73EKVGMRYQ, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.525935922 (21h9m13.565509042s) [noise=true , address=45.91.116.7:4010 , piece_id=UQRAOTS6AILKBAR6QC3ZD3VATJEYM2T6T4K2KDVU3XRKYQSEB66A, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.526222790 (21h9m13.56579591s) [noise=true , address=45.91.116.13:4153 , piece_id=KUSBCYGRA2XFXZPOYJG4EYGZ7FOSKPKV2EEFQHUND6TPTNQX7AGQ, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.526424746 (21h9m13.565997866s) [noise=true , address=77.90.31.10:4518 , piece_id=XBJTZKIXGIW3WPKU4RWYRZBMAC67YYDYRNAJNBWRYX6SHU3T73WA, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:39.362042598 (21h9m15.401615708s) [upload=18635]: Next returned nil segment Sep 7 10:37:39.362059750 (21h9m15.40163287s) [upload=18635]: Waiting for error group managing segments... Sep 7 10:37:39.669685089 (21h9m15.709258209s) [noise=true , address=45.91.116.14:4191 , piece_id=BLZ4EZD5DQ7QQNNRLNCKAB2CQBZGQ2SYKGJJIOZCXNGS4SYOFNXA, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.679437981 (21h9m15.719011101s) [noise=true , address=77.90.31.10:4519 , piece_id=5RGPSBJO5NDUDD7Y4NH5CYTGF2FB5RO6ADKEP7YUGZFVMCTACVKA, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.679555526 (21h9m15.719128646s) [noise=true , address=45.91.116.12:4225 , piece_id=JQOKRSTWFE56HRHV4BBEL3ZR7T24XBHVYYU4WA4SWJGF432EPOLQ, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.687220604 (21h9m15.726793724s) [noise=true , address=45.91.116.7:4010 , piece_id=UQRAOTS6AILKBAR6QC3ZD3VATJEYM2T6T4K2KDVU3XRKYQSEB66A, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.711092863 (21h9m15.750665983s) [noise=true , address=45.91.116.4:4939 , piece_id=BXLXHF3W5CPAS73RZXC4GL4PPN5PRMVLSSW4ADSW2M2KWRXQY77Q, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.734404882 (21h9m15.773978002s) [noise=true , address=45.91.116.7:4009 , piece_id=TYKXIJX7XIMFYKEA5RMORUFVO7UDVKNVBSJAIFUEIB3A76G4HN3Q, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.743143673 (21h9m15.782716793s) [noise=true , address=45.91.116.11:4119 , piece_id=Y5TR5QSARCJ3HXQZN6OWO7NTLMSGCSLCOEZVJRSB3VZQQL7O57NA, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.748574906 (21h9m15.788148016s) [noise=true , address=45.91.116.6:4974 , piece_id=ZUB4DD6ZXILUSYESNMGNOCKZVTPUY24N52YF2KY3Z36JYNDGRM2A, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.754311293 (21h9m15.793884413s) [noise=true , address=45.91.116.13:4153 , piece_id=KUSBCYGRA2XFXZPOYJG4EYGZ7FOSKPKV2EEFQHUND6TPTNQX7AGQ, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.755384276 (21h9m15.794957396s) [noise=true , address=45.91.116.5:4298 , piece_id=T4KK43URSFHH2LDPBX4CN66G4KVBCWV7JSJM3CADPG6EYGOS2HFQ, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.758010662 (21h9m15.797583782s) [noise=true , address=77.90.31.10:4518 , piece_id=XBJTZKIXGIW3WPKU4RWYRZBMAC67YYDYRNAJNBWRYX6SHU3T73WA, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.774473184 (21h9m15.814046304s) [noise=true , address=45.91.116.6:4973 , piece_id=X63L2GPVONW7ORT5DYOQIHV46EG4DIHEB6TLOZMXKLCQPR5EWDFA, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.817437797 (21h9m15.857010917s) [noise=true , address=45.91.116.11:4117 , piece_id=5E2JPHN5T2SETJNOGGAESLRU7PRPUCUDEO25QDW6PA73EKVGMRYQ, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.817462975 (21h9m15.857036096s) [seg_pos={0 0}, upload=18635]: Segment reached optimal threshold of 13 pieces. Sep 7 10:37:39.817597944 (21h9m15.857171064s) [noise=true , address=45.91.116.6:4975 , piece_id=7U2D53SCSJ5GUGLFWEH6FA66Q5U7EIK7YVBTP5GP5P5OOAVOHQWA, seg_pos={0 0}, upload=18635]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: expected piece hash; context canceled; EOF Sep 7 10:37:39.817683578 (21h9m15.857256698s) [noise=true , address=45.91.116.10:4083 , piece_id=2XGLF7IS7LC7TU5X2M27UEYQBVX3UML3UEWSFUISJOXF6AVKITTQ, seg_pos={0 0}, upload=18635]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: expected piece hash; context canceled; EOF Sep 7 10:37:39.817775434 (21h9m15.857348554s) [noise=true , address=45.91.116.12:4227 , piece_id=7BZ46KAIEIJDIKHXRLDCMB4Y6E7KSC4QLBOLY2A2NZFA4GVVXD2A, seg_pos={0 0}, upload=18635]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH): protocol: expected piece hash; context canceled; EOF Sep 7 10:37:39.817852270 (21h9m15.8574254s) [seg_pos={0 0}, upload=18635]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 7 10:37:39.817955087 (21h9m15.857528217s) [upload=18635]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 7 10:37:39.817961028 (21h9m15.857534148s) [upload=18635]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 7 10:37:40.298833319 (21h9m16.338406429s) [upload=18635]: Done uploading segments... Sep 7 10:37:40.298849160 (21h9m16.33842228s) [upload=18635]: Upload finished. err: <nil> Duration: 7.434174712s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................X..................] duration: 10.172070688s FAIL (2025-09-07 10:52:10 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Sep 7 10:37:37.259810434 (21h9m13.299383554s) [upload=18635]: Starting upload Sep 7 10:37:37.259869848 (21h9m13.299442968s) [upload=18635]: Done starting upload Sep 7 10:37:37.259991862 (21h9m13.299564982s) [upload=18635]: Uploading segments... Sep 7 10:37:37.260091512 (21h9m13.299664632s) [upload=18635]: Got next segment. Inline: false Sep 7 10:37:37.260124284 (21h9m13.299697405s) [upload=18635]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 7 10:37:37.260129676 (21h9m13.299702796s) [upload=18635]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 7 10:37:37.523765418 (21h9m13.563338538s) [seg_pos={0 0}, upload=18635]: Begin upload segment... Sep 7 10:37:37.523848977 (21h9m13.563422098s) [seg_pos={0 0}, upload=18635]: Done begin upload segment. Sep 7 10:37:37.523875308 (21h9m13.563448418s) [noise=true , address=45.91.116.11:4119 , piece_id=Y5TR5QSARCJ3HXQZN6OWO7NTLMSGCSLCOEZVJRSB3VZQQL7O57NA, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.523927959 (21h9m13.563501079s) [noise=true , address=45.91.116.5:4298 , piece_id=T4KK43URSFHH2LDPBX4CN66G4KVBCWV7JSJM3CADPG6EYGOS2HFQ, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.524262138 (21h9m13.563835258s) [noise=true , address=45.91.116.6:4973 , piece_id=X63L2GPVONW7ORT5DYOQIHV46EG4DIHEB6TLOZMXKLCQPR5EWDFA, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.524469726 (21h9m13.564042846s) [noise=true , address=45.91.116.6:4974 , piece_id=ZUB4DD6ZXILUSYESNMGNOCKZVTPUY24N52YF2KY3Z36JYNDGRM2A, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.524553666 (21h9m13.564126786s) [noise=true , address=45.91.116.4:4939 , piece_id=BXLXHF3W5CPAS73RZXC4GL4PPN5PRMVLSSW4ADSW2M2KWRXQY77Q, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.524760001 (21h9m13.564333121s) [noise=true , address=77.90.31.10:4519 , piece_id=5RGPSBJO5NDUDD7Y4NH5CYTGF2FB5RO6ADKEP7YUGZFVMCTACVKA, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.524839723 (21h9m13.564412843s) [noise=true , address=45.91.116.12:4225 , piece_id=JQOKRSTWFE56HRHV4BBEL3ZR7T24XBHVYYU4WA4SWJGF432EPOLQ, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.525132163 (21h9m13.564705293s) [noise=true , address=45.91.116.12:4227 , piece_id=7BZ46KAIEIJDIKHXRLDCMB4Y6E7KSC4QLBOLY2A2NZFA4GVVXD2A, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.525296006 (21h9m13.564869127s) [noise=true , address=45.91.116.6:4975 , piece_id=7U2D53SCSJ5GUGLFWEH6FA66Q5U7EIK7YVBTP5GP5P5OOAVOHQWA, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.525386890 (21h9m13.56496001s) [noise=true , address=45.91.116.10:4083 , piece_id=2XGLF7IS7LC7TU5X2M27UEYQBVX3UML3UEWSFUISJOXF6AVKITTQ, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.525611531 (21h9m13.565184651s) [noise=true , address=45.91.116.7:4009 , piece_id=TYKXIJX7XIMFYKEA5RMORUFVO7UDVKNVBSJAIFUEIB3A76G4HN3Q, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.525648361 (21h9m13.565221481s) [noise=true , address=45.91.116.14:4191 , piece_id=BLZ4EZD5DQ7QQNNRLNCKAB2CQBZGQ2SYKGJJIOZCXNGS4SYOFNXA, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.525870506 (21h9m13.565443626s) [noise=true , address=45.91.116.11:4117 , piece_id=5E2JPHN5T2SETJNOGGAESLRU7PRPUCUDEO25QDW6PA73EKVGMRYQ, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.525935922 (21h9m13.565509042s) [noise=true , address=45.91.116.7:4010 , piece_id=UQRAOTS6AILKBAR6QC3ZD3VATJEYM2T6T4K2KDVU3XRKYQSEB66A, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.526222790 (21h9m13.56579591s) [noise=true , address=45.91.116.13:4153 , piece_id=KUSBCYGRA2XFXZPOYJG4EYGZ7FOSKPKV2EEFQHUND6TPTNQX7AGQ, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.526424746 (21h9m13.565997866s) [noise=true , address=77.90.31.10:4518 , piece_id=XBJTZKIXGIW3WPKU4RWYRZBMAC67YYDYRNAJNBWRYX6SHU3T73WA, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:39.362042598 (21h9m15.401615708s) [upload=18635]: Next returned nil segment Sep 7 10:37:39.362059750 (21h9m15.40163287s) [upload=18635]: Waiting for error group managing segments... Sep 7 10:37:39.669685089 (21h9m15.709258209s) [noise=true , address=45.91.116.14:4191 , piece_id=BLZ4EZD5DQ7QQNNRLNCKAB2CQBZGQ2SYKGJJIOZCXNGS4SYOFNXA, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.679437981 (21h9m15.719011101s) [noise=true , address=77.90.31.10:4519 , piece_id=5RGPSBJO5NDUDD7Y4NH5CYTGF2FB5RO6ADKEP7YUGZFVMCTACVKA, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.679555526 (21h9m15.719128646s) [noise=true , address=45.91.116.12:4225 , piece_id=JQOKRSTWFE56HRHV4BBEL3ZR7T24XBHVYYU4WA4SWJGF432EPOLQ, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.687220604 (21h9m15.726793724s) [noise=true , address=45.91.116.7:4010 , piece_id=UQRAOTS6AILKBAR6QC3ZD3VATJEYM2T6T4K2KDVU3XRKYQSEB66A, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.711092863 (21h9m15.750665983s) [noise=true , address=45.91.116.4:4939 , piece_id=BXLXHF3W5CPAS73RZXC4GL4PPN5PRMVLSSW4ADSW2M2KWRXQY77Q, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.734404882 (21h9m15.773978002s) [noise=true , address=45.91.116.7:4009 , piece_id=TYKXIJX7XIMFYKEA5RMORUFVO7UDVKNVBSJAIFUEIB3A76G4HN3Q, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.743143673 (21h9m15.782716793s) [noise=true , address=45.91.116.11:4119 , piece_id=Y5TR5QSARCJ3HXQZN6OWO7NTLMSGCSLCOEZVJRSB3VZQQL7O57NA, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.748574906 (21h9m15.788148016s) [noise=true , address=45.91.116.6:4974 , piece_id=ZUB4DD6ZXILUSYESNMGNOCKZVTPUY24N52YF2KY3Z36JYNDGRM2A, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.754311293 (21h9m15.793884413s) [noise=true , address=45.91.116.13:4153 , piece_id=KUSBCYGRA2XFXZPOYJG4EYGZ7FOSKPKV2EEFQHUND6TPTNQX7AGQ, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.755384276 (21h9m15.794957396s) [noise=true , address=45.91.116.5:4298 , piece_id=T4KK43URSFHH2LDPBX4CN66G4KVBCWV7JSJM3CADPG6EYGOS2HFQ, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.758010662 (21h9m15.797583782s) [noise=true , address=77.90.31.10:4518 , piece_id=XBJTZKIXGIW3WPKU4RWYRZBMAC67YYDYRNAJNBWRYX6SHU3T73WA, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.774473184 (21h9m15.814046304s) [noise=true , address=45.91.116.6:4973 , piece_id=X63L2GPVONW7ORT5DYOQIHV46EG4DIHEB6TLOZMXKLCQPR5EWDFA, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.817437797 (21h9m15.857010917s) [noise=true , address=45.91.116.11:4117 , piece_id=5E2JPHN5T2SETJNOGGAESLRU7PRPUCUDEO25QDW6PA73EKVGMRYQ, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.817462975 (21h9m15.857036096s) [seg_pos={0 0}, upload=18635]: Segment reached optimal threshold of 13 pieces. Sep 7 10:37:39.817597944 (21h9m15.857171064s) [noise=true , address=45.91.116.6:4975 , piece_id=7U2D53SCSJ5GUGLFWEH6FA66Q5U7EIK7YVBTP5GP5P5OOAVOHQWA, seg_pos={0 0}, upload=18635]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: expected piece hash; context canceled; EOF Sep 7 10:37:39.817683578 (21h9m15.857256698s) [noise=true , address=45.91.116.10:4083 , piece_id=2XGLF7IS7LC7TU5X2M27UEYQBVX3UML3UEWSFUISJOXF6AVKITTQ, seg_pos={0 0}, upload=18635]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: expected piece hash; context canceled; EOF Sep 7 10:37:39.817775434 (21h9m15.857348554s) [noise=true , address=45.91.116.12:4227 , piece_id=7BZ46KAIEIJDIKHXRLDCMB4Y6E7KSC4QLBOLY2A2NZFA4GVVXD2A, seg_pos={0 0}, upload=18635]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH): protocol: expected piece hash; context canceled; EOF Sep 7 10:37:39.817852270 (21h9m15.8574254s) [seg_pos={0 0}, upload=18635]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 7 10:37:39.817955087 (21h9m15.857528217s) [upload=18635]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 7 10:37:39.817961028 (21h9m15.857534148s) [upload=18635]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 7 10:37:40.298833319 (21h9m16.338406429s) [upload=18635]: Done uploading segments... Sep 7 10:37:40.298849160 (21h9m16.33842228s) [upload=18635]: Upload finished. err: <nil> Duration: 10.17210318s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................X.................] duration: 8.43409149s FAIL (2025-09-07 10:51:18 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Sep 7 10:37:37.259810434 (21h9m13.299383554s) [upload=18635]: Starting upload Sep 7 10:37:37.259869848 (21h9m13.299442968s) [upload=18635]: Done starting upload Sep 7 10:37:37.259991862 (21h9m13.299564982s) [upload=18635]: Uploading segments... Sep 7 10:37:37.260091512 (21h9m13.299664632s) [upload=18635]: Got next segment. Inline: false Sep 7 10:37:37.260124284 (21h9m13.299697405s) [upload=18635]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 7 10:37:37.260129676 (21h9m13.299702796s) [upload=18635]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 7 10:37:37.523765418 (21h9m13.563338538s) [seg_pos={0 0}, upload=18635]: Begin upload segment... Sep 7 10:37:37.523848977 (21h9m13.563422098s) [seg_pos={0 0}, upload=18635]: Done begin upload segment. Sep 7 10:37:37.523875308 (21h9m13.563448418s) [noise=true , address=45.91.116.11:4119 , piece_id=Y5TR5QSARCJ3HXQZN6OWO7NTLMSGCSLCOEZVJRSB3VZQQL7O57NA, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.523927959 (21h9m13.563501079s) [noise=true , address=45.91.116.5:4298 , piece_id=T4KK43URSFHH2LDPBX4CN66G4KVBCWV7JSJM3CADPG6EYGOS2HFQ, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.524262138 (21h9m13.563835258s) [noise=true , address=45.91.116.6:4973 , piece_id=X63L2GPVONW7ORT5DYOQIHV46EG4DIHEB6TLOZMXKLCQPR5EWDFA, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.524469726 (21h9m13.564042846s) [noise=true , address=45.91.116.6:4974 , piece_id=ZUB4DD6ZXILUSYESNMGNOCKZVTPUY24N52YF2KY3Z36JYNDGRM2A, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.524553666 (21h9m13.564126786s) [noise=true , address=45.91.116.4:4939 , piece_id=BXLXHF3W5CPAS73RZXC4GL4PPN5PRMVLSSW4ADSW2M2KWRXQY77Q, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.524760001 (21h9m13.564333121s) [noise=true , address=77.90.31.10:4519 , piece_id=5RGPSBJO5NDUDD7Y4NH5CYTGF2FB5RO6ADKEP7YUGZFVMCTACVKA, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.524839723 (21h9m13.564412843s) [noise=true , address=45.91.116.12:4225 , piece_id=JQOKRSTWFE56HRHV4BBEL3ZR7T24XBHVYYU4WA4SWJGF432EPOLQ, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.525132163 (21h9m13.564705293s) [noise=true , address=45.91.116.12:4227 , piece_id=7BZ46KAIEIJDIKHXRLDCMB4Y6E7KSC4QLBOLY2A2NZFA4GVVXD2A, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.525296006 (21h9m13.564869127s) [noise=true , address=45.91.116.6:4975 , piece_id=7U2D53SCSJ5GUGLFWEH6FA66Q5U7EIK7YVBTP5GP5P5OOAVOHQWA, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.525386890 (21h9m13.56496001s) [noise=true , address=45.91.116.10:4083 , piece_id=2XGLF7IS7LC7TU5X2M27UEYQBVX3UML3UEWSFUISJOXF6AVKITTQ, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.525611531 (21h9m13.565184651s) [noise=true , address=45.91.116.7:4009 , piece_id=TYKXIJX7XIMFYKEA5RMORUFVO7UDVKNVBSJAIFUEIB3A76G4HN3Q, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.525648361 (21h9m13.565221481s) [noise=true , address=45.91.116.14:4191 , piece_id=BLZ4EZD5DQ7QQNNRLNCKAB2CQBZGQ2SYKGJJIOZCXNGS4SYOFNXA, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.525870506 (21h9m13.565443626s) [noise=true , address=45.91.116.11:4117 , piece_id=5E2JPHN5T2SETJNOGGAESLRU7PRPUCUDEO25QDW6PA73EKVGMRYQ, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.525935922 (21h9m13.565509042s) [noise=true , address=45.91.116.7:4010 , piece_id=UQRAOTS6AILKBAR6QC3ZD3VATJEYM2T6T4K2KDVU3XRKYQSEB66A, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.526222790 (21h9m13.56579591s) [noise=true , address=45.91.116.13:4153 , piece_id=KUSBCYGRA2XFXZPOYJG4EYGZ7FOSKPKV2EEFQHUND6TPTNQX7AGQ, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.526424746 (21h9m13.565997866s) [noise=true , address=77.90.31.10:4518 , piece_id=XBJTZKIXGIW3WPKU4RWYRZBMAC67YYDYRNAJNBWRYX6SHU3T73WA, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:39.362042598 (21h9m15.401615708s) [upload=18635]: Next returned nil segment Sep 7 10:37:39.362059750 (21h9m15.40163287s) [upload=18635]: Waiting for error group managing segments... Sep 7 10:37:39.669685089 (21h9m15.709258209s) [noise=true , address=45.91.116.14:4191 , piece_id=BLZ4EZD5DQ7QQNNRLNCKAB2CQBZGQ2SYKGJJIOZCXNGS4SYOFNXA, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.679437981 (21h9m15.719011101s) [noise=true , address=77.90.31.10:4519 , piece_id=5RGPSBJO5NDUDD7Y4NH5CYTGF2FB5RO6ADKEP7YUGZFVMCTACVKA, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.679555526 (21h9m15.719128646s) [noise=true , address=45.91.116.12:4225 , piece_id=JQOKRSTWFE56HRHV4BBEL3ZR7T24XBHVYYU4WA4SWJGF432EPOLQ, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.687220604 (21h9m15.726793724s) [noise=true , address=45.91.116.7:4010 , piece_id=UQRAOTS6AILKBAR6QC3ZD3VATJEYM2T6T4K2KDVU3XRKYQSEB66A, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.711092863 (21h9m15.750665983s) [noise=true , address=45.91.116.4:4939 , piece_id=BXLXHF3W5CPAS73RZXC4GL4PPN5PRMVLSSW4ADSW2M2KWRXQY77Q, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.734404882 (21h9m15.773978002s) [noise=true , address=45.91.116.7:4009 , piece_id=TYKXIJX7XIMFYKEA5RMORUFVO7UDVKNVBSJAIFUEIB3A76G4HN3Q, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.743143673 (21h9m15.782716793s) [noise=true , address=45.91.116.11:4119 , piece_id=Y5TR5QSARCJ3HXQZN6OWO7NTLMSGCSLCOEZVJRSB3VZQQL7O57NA, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.748574906 (21h9m15.788148016s) [noise=true , address=45.91.116.6:4974 , piece_id=ZUB4DD6ZXILUSYESNMGNOCKZVTPUY24N52YF2KY3Z36JYNDGRM2A, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.754311293 (21h9m15.793884413s) [noise=true , address=45.91.116.13:4153 , piece_id=KUSBCYGRA2XFXZPOYJG4EYGZ7FOSKPKV2EEFQHUND6TPTNQX7AGQ, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.755384276 (21h9m15.794957396s) [noise=true , address=45.91.116.5:4298 , piece_id=T4KK43URSFHH2LDPBX4CN66G4KVBCWV7JSJM3CADPG6EYGOS2HFQ, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.758010662 (21h9m15.797583782s) [noise=true , address=77.90.31.10:4518 , piece_id=XBJTZKIXGIW3WPKU4RWYRZBMAC67YYDYRNAJNBWRYX6SHU3T73WA, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.774473184 (21h9m15.814046304s) [noise=true , address=45.91.116.6:4973 , piece_id=X63L2GPVONW7ORT5DYOQIHV46EG4DIHEB6TLOZMXKLCQPR5EWDFA, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.817437797 (21h9m15.857010917s) [noise=true , address=45.91.116.11:4117 , piece_id=5E2JPHN5T2SETJNOGGAESLRU7PRPUCUDEO25QDW6PA73EKVGMRYQ, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.817462975 (21h9m15.857036096s) [seg_pos={0 0}, upload=18635]: Segment reached optimal threshold of 13 pieces. Sep 7 10:37:39.817597944 (21h9m15.857171064s) [noise=true , address=45.91.116.6:4975 , piece_id=7U2D53SCSJ5GUGLFWEH6FA66Q5U7EIK7YVBTP5GP5P5OOAVOHQWA, seg_pos={0 0}, upload=18635]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: expected piece hash; context canceled; EOF Sep 7 10:37:39.817683578 (21h9m15.857256698s) [noise=true , address=45.91.116.10:4083 , piece_id=2XGLF7IS7LC7TU5X2M27UEYQBVX3UML3UEWSFUISJOXF6AVKITTQ, seg_pos={0 0}, upload=18635]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: expected piece hash; context canceled; EOF Sep 7 10:37:39.817775434 (21h9m15.857348554s) [noise=true , address=45.91.116.12:4227 , piece_id=7BZ46KAIEIJDIKHXRLDCMB4Y6E7KSC4QLBOLY2A2NZFA4GVVXD2A, seg_pos={0 0}, upload=18635]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH): protocol: expected piece hash; context canceled; EOF Sep 7 10:37:39.817852270 (21h9m15.8574254s) [seg_pos={0 0}, upload=18635]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 7 10:37:39.817955087 (21h9m15.857528217s) [upload=18635]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 7 10:37:39.817961028 (21h9m15.857534148s) [upload=18635]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 7 10:37:40.298833319 (21h9m16.338406429s) [upload=18635]: Done uploading segments... Sep 7 10:37:40.298849160 (21h9m16.33842228s) [upload=18635]: Upload finished. err: <nil> Duration: 8.43413315s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................X................] duration: 9.074609237s FAIL (2025-09-07 10:50:33 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Sep 7 10:37:37.259810434 (21h9m13.299383554s) [upload=18635]: Starting upload Sep 7 10:37:37.259869848 (21h9m13.299442968s) [upload=18635]: Done starting upload Sep 7 10:37:37.259991862 (21h9m13.299564982s) [upload=18635]: Uploading segments... Sep 7 10:37:37.260091512 (21h9m13.299664632s) [upload=18635]: Got next segment. Inline: false Sep 7 10:37:37.260124284 (21h9m13.299697405s) [upload=18635]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 7 10:37:37.260129676 (21h9m13.299702796s) [upload=18635]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 7 10:37:37.523765418 (21h9m13.563338538s) [seg_pos={0 0}, upload=18635]: Begin upload segment... Sep 7 10:37:37.523848977 (21h9m13.563422098s) [seg_pos={0 0}, upload=18635]: Done begin upload segment. Sep 7 10:37:37.523875308 (21h9m13.563448418s) [noise=true , address=45.91.116.11:4119 , piece_id=Y5TR5QSARCJ3HXQZN6OWO7NTLMSGCSLCOEZVJRSB3VZQQL7O57NA, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.523927959 (21h9m13.563501079s) [noise=true , address=45.91.116.5:4298 , piece_id=T4KK43URSFHH2LDPBX4CN66G4KVBCWV7JSJM3CADPG6EYGOS2HFQ, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.524262138 (21h9m13.563835258s) [noise=true , address=45.91.116.6:4973 , piece_id=X63L2GPVONW7ORT5DYOQIHV46EG4DIHEB6TLOZMXKLCQPR5EWDFA, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.524469726 (21h9m13.564042846s) [noise=true , address=45.91.116.6:4974 , piece_id=ZUB4DD6ZXILUSYESNMGNOCKZVTPUY24N52YF2KY3Z36JYNDGRM2A, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.524553666 (21h9m13.564126786s) [noise=true , address=45.91.116.4:4939 , piece_id=BXLXHF3W5CPAS73RZXC4GL4PPN5PRMVLSSW4ADSW2M2KWRXQY77Q, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.524760001 (21h9m13.564333121s) [noise=true , address=77.90.31.10:4519 , piece_id=5RGPSBJO5NDUDD7Y4NH5CYTGF2FB5RO6ADKEP7YUGZFVMCTACVKA, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.524839723 (21h9m13.564412843s) [noise=true , address=45.91.116.12:4225 , piece_id=JQOKRSTWFE56HRHV4BBEL3ZR7T24XBHVYYU4WA4SWJGF432EPOLQ, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.525132163 (21h9m13.564705293s) [noise=true , address=45.91.116.12:4227 , piece_id=7BZ46KAIEIJDIKHXRLDCMB4Y6E7KSC4QLBOLY2A2NZFA4GVVXD2A, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.525296006 (21h9m13.564869127s) [noise=true , address=45.91.116.6:4975 , piece_id=7U2D53SCSJ5GUGLFWEH6FA66Q5U7EIK7YVBTP5GP5P5OOAVOHQWA, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.525386890 (21h9m13.56496001s) [noise=true , address=45.91.116.10:4083 , piece_id=2XGLF7IS7LC7TU5X2M27UEYQBVX3UML3UEWSFUISJOXF6AVKITTQ, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.525611531 (21h9m13.565184651s) [noise=true , address=45.91.116.7:4009 , piece_id=TYKXIJX7XIMFYKEA5RMORUFVO7UDVKNVBSJAIFUEIB3A76G4HN3Q, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.525648361 (21h9m13.565221481s) [noise=true , address=45.91.116.14:4191 , piece_id=BLZ4EZD5DQ7QQNNRLNCKAB2CQBZGQ2SYKGJJIOZCXNGS4SYOFNXA, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.525870506 (21h9m13.565443626s) [noise=true , address=45.91.116.11:4117 , piece_id=5E2JPHN5T2SETJNOGGAESLRU7PRPUCUDEO25QDW6PA73EKVGMRYQ, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.525935922 (21h9m13.565509042s) [noise=true , address=45.91.116.7:4010 , piece_id=UQRAOTS6AILKBAR6QC3ZD3VATJEYM2T6T4K2KDVU3XRKYQSEB66A, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.526222790 (21h9m13.56579591s) [noise=true , address=45.91.116.13:4153 , piece_id=KUSBCYGRA2XFXZPOYJG4EYGZ7FOSKPKV2EEFQHUND6TPTNQX7AGQ, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.526424746 (21h9m13.565997866s) [noise=true , address=77.90.31.10:4518 , piece_id=XBJTZKIXGIW3WPKU4RWYRZBMAC67YYDYRNAJNBWRYX6SHU3T73WA, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:39.362042598 (21h9m15.401615708s) [upload=18635]: Next returned nil segment Sep 7 10:37:39.362059750 (21h9m15.40163287s) [upload=18635]: Waiting for error group managing segments... Sep 7 10:37:39.669685089 (21h9m15.709258209s) [noise=true , address=45.91.116.14:4191 , piece_id=BLZ4EZD5DQ7QQNNRLNCKAB2CQBZGQ2SYKGJJIOZCXNGS4SYOFNXA, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.679437981 (21h9m15.719011101s) [noise=true , address=77.90.31.10:4519 , piece_id=5RGPSBJO5NDUDD7Y4NH5CYTGF2FB5RO6ADKEP7YUGZFVMCTACVKA, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.679555526 (21h9m15.719128646s) [noise=true , address=45.91.116.12:4225 , piece_id=JQOKRSTWFE56HRHV4BBEL3ZR7T24XBHVYYU4WA4SWJGF432EPOLQ, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.687220604 (21h9m15.726793724s) [noise=true , address=45.91.116.7:4010 , piece_id=UQRAOTS6AILKBAR6QC3ZD3VATJEYM2T6T4K2KDVU3XRKYQSEB66A, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.711092863 (21h9m15.750665983s) [noise=true , address=45.91.116.4:4939 , piece_id=BXLXHF3W5CPAS73RZXC4GL4PPN5PRMVLSSW4ADSW2M2KWRXQY77Q, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.734404882 (21h9m15.773978002s) [noise=true , address=45.91.116.7:4009 , piece_id=TYKXIJX7XIMFYKEA5RMORUFVO7UDVKNVBSJAIFUEIB3A76G4HN3Q, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.743143673 (21h9m15.782716793s) [noise=true , address=45.91.116.11:4119 , piece_id=Y5TR5QSARCJ3HXQZN6OWO7NTLMSGCSLCOEZVJRSB3VZQQL7O57NA, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.748574906 (21h9m15.788148016s) [noise=true , address=45.91.116.6:4974 , piece_id=ZUB4DD6ZXILUSYESNMGNOCKZVTPUY24N52YF2KY3Z36JYNDGRM2A, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.754311293 (21h9m15.793884413s) [noise=true , address=45.91.116.13:4153 , piece_id=KUSBCYGRA2XFXZPOYJG4EYGZ7FOSKPKV2EEFQHUND6TPTNQX7AGQ, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.755384276 (21h9m15.794957396s) [noise=true , address=45.91.116.5:4298 , piece_id=T4KK43URSFHH2LDPBX4CN66G4KVBCWV7JSJM3CADPG6EYGOS2HFQ, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.758010662 (21h9m15.797583782s) [noise=true , address=77.90.31.10:4518 , piece_id=XBJTZKIXGIW3WPKU4RWYRZBMAC67YYDYRNAJNBWRYX6SHU3T73WA, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.774473184 (21h9m15.814046304s) [noise=true , address=45.91.116.6:4973 , piece_id=X63L2GPVONW7ORT5DYOQIHV46EG4DIHEB6TLOZMXKLCQPR5EWDFA, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.817437797 (21h9m15.857010917s) [noise=true , address=45.91.116.11:4117 , piece_id=5E2JPHN5T2SETJNOGGAESLRU7PRPUCUDEO25QDW6PA73EKVGMRYQ, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.817462975 (21h9m15.857036096s) [seg_pos={0 0}, upload=18635]: Segment reached optimal threshold of 13 pieces. Sep 7 10:37:39.817597944 (21h9m15.857171064s) [noise=true , address=45.91.116.6:4975 , piece_id=7U2D53SCSJ5GUGLFWEH6FA66Q5U7EIK7YVBTP5GP5P5OOAVOHQWA, seg_pos={0 0}, upload=18635]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: expected piece hash; context canceled; EOF Sep 7 10:37:39.817683578 (21h9m15.857256698s) [noise=true , address=45.91.116.10:4083 , piece_id=2XGLF7IS7LC7TU5X2M27UEYQBVX3UML3UEWSFUISJOXF6AVKITTQ, seg_pos={0 0}, upload=18635]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: expected piece hash; context canceled; EOF Sep 7 10:37:39.817775434 (21h9m15.857348554s) [noise=true , address=45.91.116.12:4227 , piece_id=7BZ46KAIEIJDIKHXRLDCMB4Y6E7KSC4QLBOLY2A2NZFA4GVVXD2A, seg_pos={0 0}, upload=18635]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH): protocol: expected piece hash; context canceled; EOF Sep 7 10:37:39.817852270 (21h9m15.8574254s) [seg_pos={0 0}, upload=18635]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 7 10:37:39.817955087 (21h9m15.857528217s) [upload=18635]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 7 10:37:39.817961028 (21h9m15.857534148s) [upload=18635]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 7 10:37:40.298833319 (21h9m16.338406429s) [upload=18635]: Done uploading segments... Sep 7 10:37:40.298849160 (21h9m16.33842228s) [upload=18635]: Upload finished. err: <nil> Duration: 9.074637522s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................................X...............] duration: 10.614439372s FAIL (2025-09-07 10:49:56 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Sep 7 10:37:37.259810434 (21h9m13.299383554s) [upload=18635]: Starting upload Sep 7 10:37:37.259869848 (21h9m13.299442968s) [upload=18635]: Done starting upload Sep 7 10:37:37.259991862 (21h9m13.299564982s) [upload=18635]: Uploading segments... Sep 7 10:37:37.260091512 (21h9m13.299664632s) [upload=18635]: Got next segment. Inline: false Sep 7 10:37:37.260124284 (21h9m13.299697405s) [upload=18635]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 7 10:37:37.260129676 (21h9m13.299702796s) [upload=18635]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 7 10:37:37.523765418 (21h9m13.563338538s) [seg_pos={0 0}, upload=18635]: Begin upload segment... Sep 7 10:37:37.523848977 (21h9m13.563422098s) [seg_pos={0 0}, upload=18635]: Done begin upload segment. Sep 7 10:37:37.523875308 (21h9m13.563448418s) [noise=true , address=45.91.116.11:4119 , piece_id=Y5TR5QSARCJ3HXQZN6OWO7NTLMSGCSLCOEZVJRSB3VZQQL7O57NA, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.523927959 (21h9m13.563501079s) [noise=true , address=45.91.116.5:4298 , piece_id=T4KK43URSFHH2LDPBX4CN66G4KVBCWV7JSJM3CADPG6EYGOS2HFQ, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.524262138 (21h9m13.563835258s) [noise=true , address=45.91.116.6:4973 , piece_id=X63L2GPVONW7ORT5DYOQIHV46EG4DIHEB6TLOZMXKLCQPR5EWDFA, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.524469726 (21h9m13.564042846s) [noise=true , address=45.91.116.6:4974 , piece_id=ZUB4DD6ZXILUSYESNMGNOCKZVTPUY24N52YF2KY3Z36JYNDGRM2A, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.524553666 (21h9m13.564126786s) [noise=true , address=45.91.116.4:4939 , piece_id=BXLXHF3W5CPAS73RZXC4GL4PPN5PRMVLSSW4ADSW2M2KWRXQY77Q, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.524760001 (21h9m13.564333121s) [noise=true , address=77.90.31.10:4519 , piece_id=5RGPSBJO5NDUDD7Y4NH5CYTGF2FB5RO6ADKEP7YUGZFVMCTACVKA, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.524839723 (21h9m13.564412843s) [noise=true , address=45.91.116.12:4225 , piece_id=JQOKRSTWFE56HRHV4BBEL3ZR7T24XBHVYYU4WA4SWJGF432EPOLQ, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.525132163 (21h9m13.564705293s) [noise=true , address=45.91.116.12:4227 , piece_id=7BZ46KAIEIJDIKHXRLDCMB4Y6E7KSC4QLBOLY2A2NZFA4GVVXD2A, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.525296006 (21h9m13.564869127s) [noise=true , address=45.91.116.6:4975 , piece_id=7U2D53SCSJ5GUGLFWEH6FA66Q5U7EIK7YVBTP5GP5P5OOAVOHQWA, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.525386890 (21h9m13.56496001s) [noise=true , address=45.91.116.10:4083 , piece_id=2XGLF7IS7LC7TU5X2M27UEYQBVX3UML3UEWSFUISJOXF6AVKITTQ, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.525611531 (21h9m13.565184651s) [noise=true , address=45.91.116.7:4009 , piece_id=TYKXIJX7XIMFYKEA5RMORUFVO7UDVKNVBSJAIFUEIB3A76G4HN3Q, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.525648361 (21h9m13.565221481s) [noise=true , address=45.91.116.14:4191 , piece_id=BLZ4EZD5DQ7QQNNRLNCKAB2CQBZGQ2SYKGJJIOZCXNGS4SYOFNXA, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.525870506 (21h9m13.565443626s) [noise=true , address=45.91.116.11:4117 , piece_id=5E2JPHN5T2SETJNOGGAESLRU7PRPUCUDEO25QDW6PA73EKVGMRYQ, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.525935922 (21h9m13.565509042s) [noise=true , address=45.91.116.7:4010 , piece_id=UQRAOTS6AILKBAR6QC3ZD3VATJEYM2T6T4K2KDVU3XRKYQSEB66A, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.526222790 (21h9m13.56579591s) [noise=true , address=45.91.116.13:4153 , piece_id=KUSBCYGRA2XFXZPOYJG4EYGZ7FOSKPKV2EEFQHUND6TPTNQX7AGQ, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.526424746 (21h9m13.565997866s) [noise=true , address=77.90.31.10:4518 , piece_id=XBJTZKIXGIW3WPKU4RWYRZBMAC67YYDYRNAJNBWRYX6SHU3T73WA, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:39.362042598 (21h9m15.401615708s) [upload=18635]: Next returned nil segment Sep 7 10:37:39.362059750 (21h9m15.40163287s) [upload=18635]: Waiting for error group managing segments... Sep 7 10:37:39.669685089 (21h9m15.709258209s) [noise=true , address=45.91.116.14:4191 , piece_id=BLZ4EZD5DQ7QQNNRLNCKAB2CQBZGQ2SYKGJJIOZCXNGS4SYOFNXA, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.679437981 (21h9m15.719011101s) [noise=true , address=77.90.31.10:4519 , piece_id=5RGPSBJO5NDUDD7Y4NH5CYTGF2FB5RO6ADKEP7YUGZFVMCTACVKA, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.679555526 (21h9m15.719128646s) [noise=true , address=45.91.116.12:4225 , piece_id=JQOKRSTWFE56HRHV4BBEL3ZR7T24XBHVYYU4WA4SWJGF432EPOLQ, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.687220604 (21h9m15.726793724s) [noise=true , address=45.91.116.7:4010 , piece_id=UQRAOTS6AILKBAR6QC3ZD3VATJEYM2T6T4K2KDVU3XRKYQSEB66A, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.711092863 (21h9m15.750665983s) [noise=true , address=45.91.116.4:4939 , piece_id=BXLXHF3W5CPAS73RZXC4GL4PPN5PRMVLSSW4ADSW2M2KWRXQY77Q, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.734404882 (21h9m15.773978002s) [noise=true , address=45.91.116.7:4009 , piece_id=TYKXIJX7XIMFYKEA5RMORUFVO7UDVKNVBSJAIFUEIB3A76G4HN3Q, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.743143673 (21h9m15.782716793s) [noise=true , address=45.91.116.11:4119 , piece_id=Y5TR5QSARCJ3HXQZN6OWO7NTLMSGCSLCOEZVJRSB3VZQQL7O57NA, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.748574906 (21h9m15.788148016s) [noise=true , address=45.91.116.6:4974 , piece_id=ZUB4DD6ZXILUSYESNMGNOCKZVTPUY24N52YF2KY3Z36JYNDGRM2A, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.754311293 (21h9m15.793884413s) [noise=true , address=45.91.116.13:4153 , piece_id=KUSBCYGRA2XFXZPOYJG4EYGZ7FOSKPKV2EEFQHUND6TPTNQX7AGQ, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.755384276 (21h9m15.794957396s) [noise=true , address=45.91.116.5:4298 , piece_id=T4KK43URSFHH2LDPBX4CN66G4KVBCWV7JSJM3CADPG6EYGOS2HFQ, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.758010662 (21h9m15.797583782s) [noise=true , address=77.90.31.10:4518 , piece_id=XBJTZKIXGIW3WPKU4RWYRZBMAC67YYDYRNAJNBWRYX6SHU3T73WA, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.774473184 (21h9m15.814046304s) [noise=true , address=45.91.116.6:4973 , piece_id=X63L2GPVONW7ORT5DYOQIHV46EG4DIHEB6TLOZMXKLCQPR5EWDFA, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.817437797 (21h9m15.857010917s) [noise=true , address=45.91.116.11:4117 , piece_id=5E2JPHN5T2SETJNOGGAESLRU7PRPUCUDEO25QDW6PA73EKVGMRYQ, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.817462975 (21h9m15.857036096s) [seg_pos={0 0}, upload=18635]: Segment reached optimal threshold of 13 pieces. Sep 7 10:37:39.817597944 (21h9m15.857171064s) [noise=true , address=45.91.116.6:4975 , piece_id=7U2D53SCSJ5GUGLFWEH6FA66Q5U7EIK7YVBTP5GP5P5OOAVOHQWA, seg_pos={0 0}, upload=18635]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: expected piece hash; context canceled; EOF Sep 7 10:37:39.817683578 (21h9m15.857256698s) [noise=true , address=45.91.116.10:4083 , piece_id=2XGLF7IS7LC7TU5X2M27UEYQBVX3UML3UEWSFUISJOXF6AVKITTQ, seg_pos={0 0}, upload=18635]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: expected piece hash; context canceled; EOF Sep 7 10:37:39.817775434 (21h9m15.857348554s) [noise=true , address=45.91.116.12:4227 , piece_id=7BZ46KAIEIJDIKHXRLDCMB4Y6E7KSC4QLBOLY2A2NZFA4GVVXD2A, seg_pos={0 0}, upload=18635]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH): protocol: expected piece hash; context canceled; EOF Sep 7 10:37:39.817852270 (21h9m15.8574254s) [seg_pos={0 0}, upload=18635]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 7 10:37:39.817955087 (21h9m15.857528217s) [upload=18635]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 7 10:37:39.817961028 (21h9m15.857534148s) [upload=18635]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 7 10:37:40.298833319 (21h9m16.338406429s) [upload=18635]: Done uploading segments... Sep 7 10:37:40.298849160 (21h9m16.33842228s) [upload=18635]: Upload finished. err: <nil> Duration: 10.614476473s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................................X..............] duration: 9.5928477s FAIL (2025-09-07 10:49:35 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Sep 7 10:37:37.259810434 (21h9m13.299383554s) [upload=18635]: Starting upload Sep 7 10:37:37.259869848 (21h9m13.299442968s) [upload=18635]: Done starting upload Sep 7 10:37:37.259991862 (21h9m13.299564982s) [upload=18635]: Uploading segments... Sep 7 10:37:37.260091512 (21h9m13.299664632s) [upload=18635]: Got next segment. Inline: false Sep 7 10:37:37.260124284 (21h9m13.299697405s) [upload=18635]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 7 10:37:37.260129676 (21h9m13.299702796s) [upload=18635]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 7 10:37:37.523765418 (21h9m13.563338538s) [seg_pos={0 0}, upload=18635]: Begin upload segment... Sep 7 10:37:37.523848977 (21h9m13.563422098s) [seg_pos={0 0}, upload=18635]: Done begin upload segment. Sep 7 10:37:37.523875308 (21h9m13.563448418s) [noise=true , address=45.91.116.11:4119 , piece_id=Y5TR5QSARCJ3HXQZN6OWO7NTLMSGCSLCOEZVJRSB3VZQQL7O57NA, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.523927959 (21h9m13.563501079s) [noise=true , address=45.91.116.5:4298 , piece_id=T4KK43URSFHH2LDPBX4CN66G4KVBCWV7JSJM3CADPG6EYGOS2HFQ, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.524262138 (21h9m13.563835258s) [noise=true , address=45.91.116.6:4973 , piece_id=X63L2GPVONW7ORT5DYOQIHV46EG4DIHEB6TLOZMXKLCQPR5EWDFA, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.524469726 (21h9m13.564042846s) [noise=true , address=45.91.116.6:4974 , piece_id=ZUB4DD6ZXILUSYESNMGNOCKZVTPUY24N52YF2KY3Z36JYNDGRM2A, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.524553666 (21h9m13.564126786s) [noise=true , address=45.91.116.4:4939 , piece_id=BXLXHF3W5CPAS73RZXC4GL4PPN5PRMVLSSW4ADSW2M2KWRXQY77Q, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.524760001 (21h9m13.564333121s) [noise=true , address=77.90.31.10:4519 , piece_id=5RGPSBJO5NDUDD7Y4NH5CYTGF2FB5RO6ADKEP7YUGZFVMCTACVKA, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.524839723 (21h9m13.564412843s) [noise=true , address=45.91.116.12:4225 , piece_id=JQOKRSTWFE56HRHV4BBEL3ZR7T24XBHVYYU4WA4SWJGF432EPOLQ, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.525132163 (21h9m13.564705293s) [noise=true , address=45.91.116.12:4227 , piece_id=7BZ46KAIEIJDIKHXRLDCMB4Y6E7KSC4QLBOLY2A2NZFA4GVVXD2A, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.525296006 (21h9m13.564869127s) [noise=true , address=45.91.116.6:4975 , piece_id=7U2D53SCSJ5GUGLFWEH6FA66Q5U7EIK7YVBTP5GP5P5OOAVOHQWA, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.525386890 (21h9m13.56496001s) [noise=true , address=45.91.116.10:4083 , piece_id=2XGLF7IS7LC7TU5X2M27UEYQBVX3UML3UEWSFUISJOXF6AVKITTQ, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.525611531 (21h9m13.565184651s) [noise=true , address=45.91.116.7:4009 , piece_id=TYKXIJX7XIMFYKEA5RMORUFVO7UDVKNVBSJAIFUEIB3A76G4HN3Q, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.525648361 (21h9m13.565221481s) [noise=true , address=45.91.116.14:4191 , piece_id=BLZ4EZD5DQ7QQNNRLNCKAB2CQBZGQ2SYKGJJIOZCXNGS4SYOFNXA, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.525870506 (21h9m13.565443626s) [noise=true , address=45.91.116.11:4117 , piece_id=5E2JPHN5T2SETJNOGGAESLRU7PRPUCUDEO25QDW6PA73EKVGMRYQ, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.525935922 (21h9m13.565509042s) [noise=true , address=45.91.116.7:4010 , piece_id=UQRAOTS6AILKBAR6QC3ZD3VATJEYM2T6T4K2KDVU3XRKYQSEB66A, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.526222790 (21h9m13.56579591s) [noise=true , address=45.91.116.13:4153 , piece_id=KUSBCYGRA2XFXZPOYJG4EYGZ7FOSKPKV2EEFQHUND6TPTNQX7AGQ, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.526424746 (21h9m13.565997866s) [noise=true , address=77.90.31.10:4518 , piece_id=XBJTZKIXGIW3WPKU4RWYRZBMAC67YYDYRNAJNBWRYX6SHU3T73WA, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:39.362042598 (21h9m15.401615708s) [upload=18635]: Next returned nil segment Sep 7 10:37:39.362059750 (21h9m15.40163287s) [upload=18635]: Waiting for error group managing segments... Sep 7 10:37:39.669685089 (21h9m15.709258209s) [noise=true , address=45.91.116.14:4191 , piece_id=BLZ4EZD5DQ7QQNNRLNCKAB2CQBZGQ2SYKGJJIOZCXNGS4SYOFNXA, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.679437981 (21h9m15.719011101s) [noise=true , address=77.90.31.10:4519 , piece_id=5RGPSBJO5NDUDD7Y4NH5CYTGF2FB5RO6ADKEP7YUGZFVMCTACVKA, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.679555526 (21h9m15.719128646s) [noise=true , address=45.91.116.12:4225 , piece_id=JQOKRSTWFE56HRHV4BBEL3ZR7T24XBHVYYU4WA4SWJGF432EPOLQ, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.687220604 (21h9m15.726793724s) [noise=true , address=45.91.116.7:4010 , piece_id=UQRAOTS6AILKBAR6QC3ZD3VATJEYM2T6T4K2KDVU3XRKYQSEB66A, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.711092863 (21h9m15.750665983s) [noise=true , address=45.91.116.4:4939 , piece_id=BXLXHF3W5CPAS73RZXC4GL4PPN5PRMVLSSW4ADSW2M2KWRXQY77Q, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.734404882 (21h9m15.773978002s) [noise=true , address=45.91.116.7:4009 , piece_id=TYKXIJX7XIMFYKEA5RMORUFVO7UDVKNVBSJAIFUEIB3A76G4HN3Q, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.743143673 (21h9m15.782716793s) [noise=true , address=45.91.116.11:4119 , piece_id=Y5TR5QSARCJ3HXQZN6OWO7NTLMSGCSLCOEZVJRSB3VZQQL7O57NA, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.748574906 (21h9m15.788148016s) [noise=true , address=45.91.116.6:4974 , piece_id=ZUB4DD6ZXILUSYESNMGNOCKZVTPUY24N52YF2KY3Z36JYNDGRM2A, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.754311293 (21h9m15.793884413s) [noise=true , address=45.91.116.13:4153 , piece_id=KUSBCYGRA2XFXZPOYJG4EYGZ7FOSKPKV2EEFQHUND6TPTNQX7AGQ, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.755384276 (21h9m15.794957396s) [noise=true , address=45.91.116.5:4298 , piece_id=T4KK43URSFHH2LDPBX4CN66G4KVBCWV7JSJM3CADPG6EYGOS2HFQ, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.758010662 (21h9m15.797583782s) [noise=true , address=77.90.31.10:4518 , piece_id=XBJTZKIXGIW3WPKU4RWYRZBMAC67YYDYRNAJNBWRYX6SHU3T73WA, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.774473184 (21h9m15.814046304s) [noise=true , address=45.91.116.6:4973 , piece_id=X63L2GPVONW7ORT5DYOQIHV46EG4DIHEB6TLOZMXKLCQPR5EWDFA, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.817437797 (21h9m15.857010917s) [noise=true , address=45.91.116.11:4117 , piece_id=5E2JPHN5T2SETJNOGGAESLRU7PRPUCUDEO25QDW6PA73EKVGMRYQ, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.817462975 (21h9m15.857036096s) [seg_pos={0 0}, upload=18635]: Segment reached optimal threshold of 13 pieces. Sep 7 10:37:39.817597944 (21h9m15.857171064s) [noise=true , address=45.91.116.6:4975 , piece_id=7U2D53SCSJ5GUGLFWEH6FA66Q5U7EIK7YVBTP5GP5P5OOAVOHQWA, seg_pos={0 0}, upload=18635]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: expected piece hash; context canceled; EOF Sep 7 10:37:39.817683578 (21h9m15.857256698s) [noise=true , address=45.91.116.10:4083 , piece_id=2XGLF7IS7LC7TU5X2M27UEYQBVX3UML3UEWSFUISJOXF6AVKITTQ, seg_pos={0 0}, upload=18635]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: expected piece hash; context canceled; EOF Sep 7 10:37:39.817775434 (21h9m15.857348554s) [noise=true , address=45.91.116.12:4227 , piece_id=7BZ46KAIEIJDIKHXRLDCMB4Y6E7KSC4QLBOLY2A2NZFA4GVVXD2A, seg_pos={0 0}, upload=18635]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH): protocol: expected piece hash; context canceled; EOF Sep 7 10:37:39.817852270 (21h9m15.8574254s) [seg_pos={0 0}, upload=18635]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 7 10:37:39.817955087 (21h9m15.857528217s) [upload=18635]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 7 10:37:39.817961028 (21h9m15.857534148s) [upload=18635]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 7 10:37:40.298833319 (21h9m16.338406429s) [upload=18635]: Done uploading segments... Sep 7 10:37:40.298849160 (21h9m16.33842228s) [upload=18635]: Upload finished. err: <nil> Duration: 9.592880994s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................................X.............] duration: 10.188835028s FAIL (2025-09-07 10:48:42 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Sep 7 10:37:37.259810434 (21h9m13.299383554s) [upload=18635]: Starting upload Sep 7 10:37:37.259869848 (21h9m13.299442968s) [upload=18635]: Done starting upload Sep 7 10:37:37.259991862 (21h9m13.299564982s) [upload=18635]: Uploading segments... Sep 7 10:37:37.260091512 (21h9m13.299664632s) [upload=18635]: Got next segment. Inline: false Sep 7 10:37:37.260124284 (21h9m13.299697405s) [upload=18635]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 7 10:37:37.260129676 (21h9m13.299702796s) [upload=18635]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 7 10:37:37.523765418 (21h9m13.563338538s) [seg_pos={0 0}, upload=18635]: Begin upload segment... Sep 7 10:37:37.523848977 (21h9m13.563422098s) [seg_pos={0 0}, upload=18635]: Done begin upload segment. Sep 7 10:37:37.523875308 (21h9m13.563448418s) [noise=true , address=45.91.116.11:4119 , piece_id=Y5TR5QSARCJ3HXQZN6OWO7NTLMSGCSLCOEZVJRSB3VZQQL7O57NA, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.523927959 (21h9m13.563501079s) [noise=true , address=45.91.116.5:4298 , piece_id=T4KK43URSFHH2LDPBX4CN66G4KVBCWV7JSJM3CADPG6EYGOS2HFQ, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.524262138 (21h9m13.563835258s) [noise=true , address=45.91.116.6:4973 , piece_id=X63L2GPVONW7ORT5DYOQIHV46EG4DIHEB6TLOZMXKLCQPR5EWDFA, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.524469726 (21h9m13.564042846s) [noise=true , address=45.91.116.6:4974 , piece_id=ZUB4DD6ZXILUSYESNMGNOCKZVTPUY24N52YF2KY3Z36JYNDGRM2A, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.524553666 (21h9m13.564126786s) [noise=true , address=45.91.116.4:4939 , piece_id=BXLXHF3W5CPAS73RZXC4GL4PPN5PRMVLSSW4ADSW2M2KWRXQY77Q, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.524760001 (21h9m13.564333121s) [noise=true , address=77.90.31.10:4519 , piece_id=5RGPSBJO5NDUDD7Y4NH5CYTGF2FB5RO6ADKEP7YUGZFVMCTACVKA, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.524839723 (21h9m13.564412843s) [noise=true , address=45.91.116.12:4225 , piece_id=JQOKRSTWFE56HRHV4BBEL3ZR7T24XBHVYYU4WA4SWJGF432EPOLQ, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.525132163 (21h9m13.564705293s) [noise=true , address=45.91.116.12:4227 , piece_id=7BZ46KAIEIJDIKHXRLDCMB4Y6E7KSC4QLBOLY2A2NZFA4GVVXD2A, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.525296006 (21h9m13.564869127s) [noise=true , address=45.91.116.6:4975 , piece_id=7U2D53SCSJ5GUGLFWEH6FA66Q5U7EIK7YVBTP5GP5P5OOAVOHQWA, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.525386890 (21h9m13.56496001s) [noise=true , address=45.91.116.10:4083 , piece_id=2XGLF7IS7LC7TU5X2M27UEYQBVX3UML3UEWSFUISJOXF6AVKITTQ, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.525611531 (21h9m13.565184651s) [noise=true , address=45.91.116.7:4009 , piece_id=TYKXIJX7XIMFYKEA5RMORUFVO7UDVKNVBSJAIFUEIB3A76G4HN3Q, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.525648361 (21h9m13.565221481s) [noise=true , address=45.91.116.14:4191 , piece_id=BLZ4EZD5DQ7QQNNRLNCKAB2CQBZGQ2SYKGJJIOZCXNGS4SYOFNXA, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.525870506 (21h9m13.565443626s) [noise=true , address=45.91.116.11:4117 , piece_id=5E2JPHN5T2SETJNOGGAESLRU7PRPUCUDEO25QDW6PA73EKVGMRYQ, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.525935922 (21h9m13.565509042s) [noise=true , address=45.91.116.7:4010 , piece_id=UQRAOTS6AILKBAR6QC3ZD3VATJEYM2T6T4K2KDVU3XRKYQSEB66A, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.526222790 (21h9m13.56579591s) [noise=true , address=45.91.116.13:4153 , piece_id=KUSBCYGRA2XFXZPOYJG4EYGZ7FOSKPKV2EEFQHUND6TPTNQX7AGQ, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.526424746 (21h9m13.565997866s) [noise=true , address=77.90.31.10:4518 , piece_id=XBJTZKIXGIW3WPKU4RWYRZBMAC67YYDYRNAJNBWRYX6SHU3T73WA, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:39.362042598 (21h9m15.401615708s) [upload=18635]: Next returned nil segment Sep 7 10:37:39.362059750 (21h9m15.40163287s) [upload=18635]: Waiting for error group managing segments... Sep 7 10:37:39.669685089 (21h9m15.709258209s) [noise=true , address=45.91.116.14:4191 , piece_id=BLZ4EZD5DQ7QQNNRLNCKAB2CQBZGQ2SYKGJJIOZCXNGS4SYOFNXA, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.679437981 (21h9m15.719011101s) [noise=true , address=77.90.31.10:4519 , piece_id=5RGPSBJO5NDUDD7Y4NH5CYTGF2FB5RO6ADKEP7YUGZFVMCTACVKA, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.679555526 (21h9m15.719128646s) [noise=true , address=45.91.116.12:4225 , piece_id=JQOKRSTWFE56HRHV4BBEL3ZR7T24XBHVYYU4WA4SWJGF432EPOLQ, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.687220604 (21h9m15.726793724s) [noise=true , address=45.91.116.7:4010 , piece_id=UQRAOTS6AILKBAR6QC3ZD3VATJEYM2T6T4K2KDVU3XRKYQSEB66A, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.711092863 (21h9m15.750665983s) [noise=true , address=45.91.116.4:4939 , piece_id=BXLXHF3W5CPAS73RZXC4GL4PPN5PRMVLSSW4ADSW2M2KWRXQY77Q, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.734404882 (21h9m15.773978002s) [noise=true , address=45.91.116.7:4009 , piece_id=TYKXIJX7XIMFYKEA5RMORUFVO7UDVKNVBSJAIFUEIB3A76G4HN3Q, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.743143673 (21h9m15.782716793s) [noise=true , address=45.91.116.11:4119 , piece_id=Y5TR5QSARCJ3HXQZN6OWO7NTLMSGCSLCOEZVJRSB3VZQQL7O57NA, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.748574906 (21h9m15.788148016s) [noise=true , address=45.91.116.6:4974 , piece_id=ZUB4DD6ZXILUSYESNMGNOCKZVTPUY24N52YF2KY3Z36JYNDGRM2A, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.754311293 (21h9m15.793884413s) [noise=true , address=45.91.116.13:4153 , piece_id=KUSBCYGRA2XFXZPOYJG4EYGZ7FOSKPKV2EEFQHUND6TPTNQX7AGQ, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.755384276 (21h9m15.794957396s) [noise=true , address=45.91.116.5:4298 , piece_id=T4KK43URSFHH2LDPBX4CN66G4KVBCWV7JSJM3CADPG6EYGOS2HFQ, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.758010662 (21h9m15.797583782s) [noise=true , address=77.90.31.10:4518 , piece_id=XBJTZKIXGIW3WPKU4RWYRZBMAC67YYDYRNAJNBWRYX6SHU3T73WA, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.774473184 (21h9m15.814046304s) [noise=true , address=45.91.116.6:4973 , piece_id=X63L2GPVONW7ORT5DYOQIHV46EG4DIHEB6TLOZMXKLCQPR5EWDFA, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.817437797 (21h9m15.857010917s) [noise=true , address=45.91.116.11:4117 , piece_id=5E2JPHN5T2SETJNOGGAESLRU7PRPUCUDEO25QDW6PA73EKVGMRYQ, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.817462975 (21h9m15.857036096s) [seg_pos={0 0}, upload=18635]: Segment reached optimal threshold of 13 pieces. Sep 7 10:37:39.817597944 (21h9m15.857171064s) [noise=true , address=45.91.116.6:4975 , piece_id=7U2D53SCSJ5GUGLFWEH6FA66Q5U7EIK7YVBTP5GP5P5OOAVOHQWA, seg_pos={0 0}, upload=18635]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: expected piece hash; context canceled; EOF Sep 7 10:37:39.817683578 (21h9m15.857256698s) [noise=true , address=45.91.116.10:4083 , piece_id=2XGLF7IS7LC7TU5X2M27UEYQBVX3UML3UEWSFUISJOXF6AVKITTQ, seg_pos={0 0}, upload=18635]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: expected piece hash; context canceled; EOF Sep 7 10:37:39.817775434 (21h9m15.857348554s) [noise=true , address=45.91.116.12:4227 , piece_id=7BZ46KAIEIJDIKHXRLDCMB4Y6E7KSC4QLBOLY2A2NZFA4GVVXD2A, seg_pos={0 0}, upload=18635]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH): protocol: expected piece hash; context canceled; EOF Sep 7 10:37:39.817852270 (21h9m15.8574254s) [seg_pos={0 0}, upload=18635]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 7 10:37:39.817955087 (21h9m15.857528217s) [upload=18635]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 7 10:37:39.817961028 (21h9m15.857534148s) [upload=18635]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 7 10:37:40.298833319 (21h9m16.338406429s) [upload=18635]: Done uploading segments... Sep 7 10:37:40.298849160 (21h9m16.33842228s) [upload=18635]: Upload finished. err: <nil> Duration: 10.188865717s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................................X............] duration: 12.644182891s FAIL (2025-09-07 10:47:30 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Sep 7 10:37:37.259810434 (21h9m13.299383554s) [upload=18635]: Starting upload Sep 7 10:37:37.259869848 (21h9m13.299442968s) [upload=18635]: Done starting upload Sep 7 10:37:37.259991862 (21h9m13.299564982s) [upload=18635]: Uploading segments... Sep 7 10:37:37.260091512 (21h9m13.299664632s) [upload=18635]: Got next segment. Inline: false Sep 7 10:37:37.260124284 (21h9m13.299697405s) [upload=18635]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 7 10:37:37.260129676 (21h9m13.299702796s) [upload=18635]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 7 10:37:37.523765418 (21h9m13.563338538s) [seg_pos={0 0}, upload=18635]: Begin upload segment... Sep 7 10:37:37.523848977 (21h9m13.563422098s) [seg_pos={0 0}, upload=18635]: Done begin upload segment. Sep 7 10:37:37.523875308 (21h9m13.563448418s) [noise=true , address=45.91.116.11:4119 , piece_id=Y5TR5QSARCJ3HXQZN6OWO7NTLMSGCSLCOEZVJRSB3VZQQL7O57NA, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.523927959 (21h9m13.563501079s) [noise=true , address=45.91.116.5:4298 , piece_id=T4KK43URSFHH2LDPBX4CN66G4KVBCWV7JSJM3CADPG6EYGOS2HFQ, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.524262138 (21h9m13.563835258s) [noise=true , address=45.91.116.6:4973 , piece_id=X63L2GPVONW7ORT5DYOQIHV46EG4DIHEB6TLOZMXKLCQPR5EWDFA, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.524469726 (21h9m13.564042846s) [noise=true , address=45.91.116.6:4974 , piece_id=ZUB4DD6ZXILUSYESNMGNOCKZVTPUY24N52YF2KY3Z36JYNDGRM2A, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.524553666 (21h9m13.564126786s) [noise=true , address=45.91.116.4:4939 , piece_id=BXLXHF3W5CPAS73RZXC4GL4PPN5PRMVLSSW4ADSW2M2KWRXQY77Q, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.524760001 (21h9m13.564333121s) [noise=true , address=77.90.31.10:4519 , piece_id=5RGPSBJO5NDUDD7Y4NH5CYTGF2FB5RO6ADKEP7YUGZFVMCTACVKA, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.524839723 (21h9m13.564412843s) [noise=true , address=45.91.116.12:4225 , piece_id=JQOKRSTWFE56HRHV4BBEL3ZR7T24XBHVYYU4WA4SWJGF432EPOLQ, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.525132163 (21h9m13.564705293s) [noise=true , address=45.91.116.12:4227 , piece_id=7BZ46KAIEIJDIKHXRLDCMB4Y6E7KSC4QLBOLY2A2NZFA4GVVXD2A, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.525296006 (21h9m13.564869127s) [noise=true , address=45.91.116.6:4975 , piece_id=7U2D53SCSJ5GUGLFWEH6FA66Q5U7EIK7YVBTP5GP5P5OOAVOHQWA, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.525386890 (21h9m13.56496001s) [noise=true , address=45.91.116.10:4083 , piece_id=2XGLF7IS7LC7TU5X2M27UEYQBVX3UML3UEWSFUISJOXF6AVKITTQ, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.525611531 (21h9m13.565184651s) [noise=true , address=45.91.116.7:4009 , piece_id=TYKXIJX7XIMFYKEA5RMORUFVO7UDVKNVBSJAIFUEIB3A76G4HN3Q, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.525648361 (21h9m13.565221481s) [noise=true , address=45.91.116.14:4191 , piece_id=BLZ4EZD5DQ7QQNNRLNCKAB2CQBZGQ2SYKGJJIOZCXNGS4SYOFNXA, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.525870506 (21h9m13.565443626s) [noise=true , address=45.91.116.11:4117 , piece_id=5E2JPHN5T2SETJNOGGAESLRU7PRPUCUDEO25QDW6PA73EKVGMRYQ, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.525935922 (21h9m13.565509042s) [noise=true , address=45.91.116.7:4010 , piece_id=UQRAOTS6AILKBAR6QC3ZD3VATJEYM2T6T4K2KDVU3XRKYQSEB66A, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.526222790 (21h9m13.56579591s) [noise=true , address=45.91.116.13:4153 , piece_id=KUSBCYGRA2XFXZPOYJG4EYGZ7FOSKPKV2EEFQHUND6TPTNQX7AGQ, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.526424746 (21h9m13.565997866s) [noise=true , address=77.90.31.10:4518 , piece_id=XBJTZKIXGIW3WPKU4RWYRZBMAC67YYDYRNAJNBWRYX6SHU3T73WA, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:39.362042598 (21h9m15.401615708s) [upload=18635]: Next returned nil segment Sep 7 10:37:39.362059750 (21h9m15.40163287s) [upload=18635]: Waiting for error group managing segments... Sep 7 10:37:39.669685089 (21h9m15.709258209s) [noise=true , address=45.91.116.14:4191 , piece_id=BLZ4EZD5DQ7QQNNRLNCKAB2CQBZGQ2SYKGJJIOZCXNGS4SYOFNXA, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.679437981 (21h9m15.719011101s) [noise=true , address=77.90.31.10:4519 , piece_id=5RGPSBJO5NDUDD7Y4NH5CYTGF2FB5RO6ADKEP7YUGZFVMCTACVKA, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.679555526 (21h9m15.719128646s) [noise=true , address=45.91.116.12:4225 , piece_id=JQOKRSTWFE56HRHV4BBEL3ZR7T24XBHVYYU4WA4SWJGF432EPOLQ, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.687220604 (21h9m15.726793724s) [noise=true , address=45.91.116.7:4010 , piece_id=UQRAOTS6AILKBAR6QC3ZD3VATJEYM2T6T4K2KDVU3XRKYQSEB66A, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.711092863 (21h9m15.750665983s) [noise=true , address=45.91.116.4:4939 , piece_id=BXLXHF3W5CPAS73RZXC4GL4PPN5PRMVLSSW4ADSW2M2KWRXQY77Q, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.734404882 (21h9m15.773978002s) [noise=true , address=45.91.116.7:4009 , piece_id=TYKXIJX7XIMFYKEA5RMORUFVO7UDVKNVBSJAIFUEIB3A76G4HN3Q, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.743143673 (21h9m15.782716793s) [noise=true , address=45.91.116.11:4119 , piece_id=Y5TR5QSARCJ3HXQZN6OWO7NTLMSGCSLCOEZVJRSB3VZQQL7O57NA, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.748574906 (21h9m15.788148016s) [noise=true , address=45.91.116.6:4974 , piece_id=ZUB4DD6ZXILUSYESNMGNOCKZVTPUY24N52YF2KY3Z36JYNDGRM2A, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.754311293 (21h9m15.793884413s) [noise=true , address=45.91.116.13:4153 , piece_id=KUSBCYGRA2XFXZPOYJG4EYGZ7FOSKPKV2EEFQHUND6TPTNQX7AGQ, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.755384276 (21h9m15.794957396s) [noise=true , address=45.91.116.5:4298 , piece_id=T4KK43URSFHH2LDPBX4CN66G4KVBCWV7JSJM3CADPG6EYGOS2HFQ, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.758010662 (21h9m15.797583782s) [noise=true , address=77.90.31.10:4518 , piece_id=XBJTZKIXGIW3WPKU4RWYRZBMAC67YYDYRNAJNBWRYX6SHU3T73WA, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.774473184 (21h9m15.814046304s) [noise=true , address=45.91.116.6:4973 , piece_id=X63L2GPVONW7ORT5DYOQIHV46EG4DIHEB6TLOZMXKLCQPR5EWDFA, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.817437797 (21h9m15.857010917s) [noise=true , address=45.91.116.11:4117 , piece_id=5E2JPHN5T2SETJNOGGAESLRU7PRPUCUDEO25QDW6PA73EKVGMRYQ, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.817462975 (21h9m15.857036096s) [seg_pos={0 0}, upload=18635]: Segment reached optimal threshold of 13 pieces. Sep 7 10:37:39.817597944 (21h9m15.857171064s) [noise=true , address=45.91.116.6:4975 , piece_id=7U2D53SCSJ5GUGLFWEH6FA66Q5U7EIK7YVBTP5GP5P5OOAVOHQWA, seg_pos={0 0}, upload=18635]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: expected piece hash; context canceled; EOF Sep 7 10:37:39.817683578 (21h9m15.857256698s) [noise=true , address=45.91.116.10:4083 , piece_id=2XGLF7IS7LC7TU5X2M27UEYQBVX3UML3UEWSFUISJOXF6AVKITTQ, seg_pos={0 0}, upload=18635]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: expected piece hash; context canceled; EOF Sep 7 10:37:39.817775434 (21h9m15.857348554s) [noise=true , address=45.91.116.12:4227 , piece_id=7BZ46KAIEIJDIKHXRLDCMB4Y6E7KSC4QLBOLY2A2NZFA4GVVXD2A, seg_pos={0 0}, upload=18635]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH): protocol: expected piece hash; context canceled; EOF Sep 7 10:37:39.817852270 (21h9m15.8574254s) [seg_pos={0 0}, upload=18635]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 7 10:37:39.817955087 (21h9m15.857528217s) [upload=18635]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 7 10:37:39.817961028 (21h9m15.857534148s) [upload=18635]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 7 10:37:40.298833319 (21h9m16.338406429s) [upload=18635]: Done uploading segments... Sep 7 10:37:40.298849160 (21h9m16.33842228s) [upload=18635]: Upload finished. err: <nil> Duration: 12.644228737s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................................X...........] duration: 8.962590286s FAIL (2025-09-07 10:46:29 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Sep 7 10:37:37.259810434 (21h9m13.299383554s) [upload=18635]: Starting upload Sep 7 10:37:37.259869848 (21h9m13.299442968s) [upload=18635]: Done starting upload Sep 7 10:37:37.259991862 (21h9m13.299564982s) [upload=18635]: Uploading segments... Sep 7 10:37:37.260091512 (21h9m13.299664632s) [upload=18635]: Got next segment. Inline: false Sep 7 10:37:37.260124284 (21h9m13.299697405s) [upload=18635]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 7 10:37:37.260129676 (21h9m13.299702796s) [upload=18635]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 7 10:37:37.523765418 (21h9m13.563338538s) [seg_pos={0 0}, upload=18635]: Begin upload segment... Sep 7 10:37:37.523848977 (21h9m13.563422098s) [seg_pos={0 0}, upload=18635]: Done begin upload segment. Sep 7 10:37:37.523875308 (21h9m13.563448418s) [noise=true , address=45.91.116.11:4119 , piece_id=Y5TR5QSARCJ3HXQZN6OWO7NTLMSGCSLCOEZVJRSB3VZQQL7O57NA, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.523927959 (21h9m13.563501079s) [noise=true , address=45.91.116.5:4298 , piece_id=T4KK43URSFHH2LDPBX4CN66G4KVBCWV7JSJM3CADPG6EYGOS2HFQ, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.524262138 (21h9m13.563835258s) [noise=true , address=45.91.116.6:4973 , piece_id=X63L2GPVONW7ORT5DYOQIHV46EG4DIHEB6TLOZMXKLCQPR5EWDFA, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.524469726 (21h9m13.564042846s) [noise=true , address=45.91.116.6:4974 , piece_id=ZUB4DD6ZXILUSYESNMGNOCKZVTPUY24N52YF2KY3Z36JYNDGRM2A, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.524553666 (21h9m13.564126786s) [noise=true , address=45.91.116.4:4939 , piece_id=BXLXHF3W5CPAS73RZXC4GL4PPN5PRMVLSSW4ADSW2M2KWRXQY77Q, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.524760001 (21h9m13.564333121s) [noise=true , address=77.90.31.10:4519 , piece_id=5RGPSBJO5NDUDD7Y4NH5CYTGF2FB5RO6ADKEP7YUGZFVMCTACVKA, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.524839723 (21h9m13.564412843s) [noise=true , address=45.91.116.12:4225 , piece_id=JQOKRSTWFE56HRHV4BBEL3ZR7T24XBHVYYU4WA4SWJGF432EPOLQ, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.525132163 (21h9m13.564705293s) [noise=true , address=45.91.116.12:4227 , piece_id=7BZ46KAIEIJDIKHXRLDCMB4Y6E7KSC4QLBOLY2A2NZFA4GVVXD2A, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.525296006 (21h9m13.564869127s) [noise=true , address=45.91.116.6:4975 , piece_id=7U2D53SCSJ5GUGLFWEH6FA66Q5U7EIK7YVBTP5GP5P5OOAVOHQWA, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.525386890 (21h9m13.56496001s) [noise=true , address=45.91.116.10:4083 , piece_id=2XGLF7IS7LC7TU5X2M27UEYQBVX3UML3UEWSFUISJOXF6AVKITTQ, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.525611531 (21h9m13.565184651s) [noise=true , address=45.91.116.7:4009 , piece_id=TYKXIJX7XIMFYKEA5RMORUFVO7UDVKNVBSJAIFUEIB3A76G4HN3Q, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.525648361 (21h9m13.565221481s) [noise=true , address=45.91.116.14:4191 , piece_id=BLZ4EZD5DQ7QQNNRLNCKAB2CQBZGQ2SYKGJJIOZCXNGS4SYOFNXA, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.525870506 (21h9m13.565443626s) [noise=true , address=45.91.116.11:4117 , piece_id=5E2JPHN5T2SETJNOGGAESLRU7PRPUCUDEO25QDW6PA73EKVGMRYQ, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.525935922 (21h9m13.565509042s) [noise=true , address=45.91.116.7:4010 , piece_id=UQRAOTS6AILKBAR6QC3ZD3VATJEYM2T6T4K2KDVU3XRKYQSEB66A, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.526222790 (21h9m13.56579591s) [noise=true , address=45.91.116.13:4153 , piece_id=KUSBCYGRA2XFXZPOYJG4EYGZ7FOSKPKV2EEFQHUND6TPTNQX7AGQ, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.526424746 (21h9m13.565997866s) [noise=true , address=77.90.31.10:4518 , piece_id=XBJTZKIXGIW3WPKU4RWYRZBMAC67YYDYRNAJNBWRYX6SHU3T73WA, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:39.362042598 (21h9m15.401615708s) [upload=18635]: Next returned nil segment Sep 7 10:37:39.362059750 (21h9m15.40163287s) [upload=18635]: Waiting for error group managing segments... Sep 7 10:37:39.669685089 (21h9m15.709258209s) [noise=true , address=45.91.116.14:4191 , piece_id=BLZ4EZD5DQ7QQNNRLNCKAB2CQBZGQ2SYKGJJIOZCXNGS4SYOFNXA, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.679437981 (21h9m15.719011101s) [noise=true , address=77.90.31.10:4519 , piece_id=5RGPSBJO5NDUDD7Y4NH5CYTGF2FB5RO6ADKEP7YUGZFVMCTACVKA, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.679555526 (21h9m15.719128646s) [noise=true , address=45.91.116.12:4225 , piece_id=JQOKRSTWFE56HRHV4BBEL3ZR7T24XBHVYYU4WA4SWJGF432EPOLQ, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.687220604 (21h9m15.726793724s) [noise=true , address=45.91.116.7:4010 , piece_id=UQRAOTS6AILKBAR6QC3ZD3VATJEYM2T6T4K2KDVU3XRKYQSEB66A, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.711092863 (21h9m15.750665983s) [noise=true , address=45.91.116.4:4939 , piece_id=BXLXHF3W5CPAS73RZXC4GL4PPN5PRMVLSSW4ADSW2M2KWRXQY77Q, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.734404882 (21h9m15.773978002s) [noise=true , address=45.91.116.7:4009 , piece_id=TYKXIJX7XIMFYKEA5RMORUFVO7UDVKNVBSJAIFUEIB3A76G4HN3Q, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.743143673 (21h9m15.782716793s) [noise=true , address=45.91.116.11:4119 , piece_id=Y5TR5QSARCJ3HXQZN6OWO7NTLMSGCSLCOEZVJRSB3VZQQL7O57NA, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.748574906 (21h9m15.788148016s) [noise=true , address=45.91.116.6:4974 , piece_id=ZUB4DD6ZXILUSYESNMGNOCKZVTPUY24N52YF2KY3Z36JYNDGRM2A, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.754311293 (21h9m15.793884413s) [noise=true , address=45.91.116.13:4153 , piece_id=KUSBCYGRA2XFXZPOYJG4EYGZ7FOSKPKV2EEFQHUND6TPTNQX7AGQ, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.755384276 (21h9m15.794957396s) [noise=true , address=45.91.116.5:4298 , piece_id=T4KK43URSFHH2LDPBX4CN66G4KVBCWV7JSJM3CADPG6EYGOS2HFQ, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.758010662 (21h9m15.797583782s) [noise=true , address=77.90.31.10:4518 , piece_id=XBJTZKIXGIW3WPKU4RWYRZBMAC67YYDYRNAJNBWRYX6SHU3T73WA, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.774473184 (21h9m15.814046304s) [noise=true , address=45.91.116.6:4973 , piece_id=X63L2GPVONW7ORT5DYOQIHV46EG4DIHEB6TLOZMXKLCQPR5EWDFA, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.817437797 (21h9m15.857010917s) [noise=true , address=45.91.116.11:4117 , piece_id=5E2JPHN5T2SETJNOGGAESLRU7PRPUCUDEO25QDW6PA73EKVGMRYQ, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.817462975 (21h9m15.857036096s) [seg_pos={0 0}, upload=18635]: Segment reached optimal threshold of 13 pieces. Sep 7 10:37:39.817597944 (21h9m15.857171064s) [noise=true , address=45.91.116.6:4975 , piece_id=7U2D53SCSJ5GUGLFWEH6FA66Q5U7EIK7YVBTP5GP5P5OOAVOHQWA, seg_pos={0 0}, upload=18635]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: expected piece hash; context canceled; EOF Sep 7 10:37:39.817683578 (21h9m15.857256698s) [noise=true , address=45.91.116.10:4083 , piece_id=2XGLF7IS7LC7TU5X2M27UEYQBVX3UML3UEWSFUISJOXF6AVKITTQ, seg_pos={0 0}, upload=18635]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: expected piece hash; context canceled; EOF Sep 7 10:37:39.817775434 (21h9m15.857348554s) [noise=true , address=45.91.116.12:4227 , piece_id=7BZ46KAIEIJDIKHXRLDCMB4Y6E7KSC4QLBOLY2A2NZFA4GVVXD2A, seg_pos={0 0}, upload=18635]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH): protocol: expected piece hash; context canceled; EOF Sep 7 10:37:39.817852270 (21h9m15.8574254s) [seg_pos={0 0}, upload=18635]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 7 10:37:39.817955087 (21h9m15.857528217s) [upload=18635]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 7 10:37:39.817961028 (21h9m15.857534148s) [upload=18635]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 7 10:37:40.298833319 (21h9m16.338406429s) [upload=18635]: Done uploading segments... Sep 7 10:37:40.298849160 (21h9m16.33842228s) [upload=18635]: Upload finished. err: <nil> Duration: 8.962621826s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................................X..........] duration: 6.263549038s FAIL (2025-09-07 10:45:20 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Sep 7 10:37:37.259810434 (21h9m13.299383554s) [upload=18635]: Starting upload Sep 7 10:37:37.259869848 (21h9m13.299442968s) [upload=18635]: Done starting upload Sep 7 10:37:37.259991862 (21h9m13.299564982s) [upload=18635]: Uploading segments... Sep 7 10:37:37.260091512 (21h9m13.299664632s) [upload=18635]: Got next segment. Inline: false Sep 7 10:37:37.260124284 (21h9m13.299697405s) [upload=18635]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 7 10:37:37.260129676 (21h9m13.299702796s) [upload=18635]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 7 10:37:37.523765418 (21h9m13.563338538s) [seg_pos={0 0}, upload=18635]: Begin upload segment... Sep 7 10:37:37.523848977 (21h9m13.563422098s) [seg_pos={0 0}, upload=18635]: Done begin upload segment. Sep 7 10:37:37.523875308 (21h9m13.563448418s) [noise=true , address=45.91.116.11:4119 , piece_id=Y5TR5QSARCJ3HXQZN6OWO7NTLMSGCSLCOEZVJRSB3VZQQL7O57NA, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.523927959 (21h9m13.563501079s) [noise=true , address=45.91.116.5:4298 , piece_id=T4KK43URSFHH2LDPBX4CN66G4KVBCWV7JSJM3CADPG6EYGOS2HFQ, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.524262138 (21h9m13.563835258s) [noise=true , address=45.91.116.6:4973 , piece_id=X63L2GPVONW7ORT5DYOQIHV46EG4DIHEB6TLOZMXKLCQPR5EWDFA, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.524469726 (21h9m13.564042846s) [noise=true , address=45.91.116.6:4974 , piece_id=ZUB4DD6ZXILUSYESNMGNOCKZVTPUY24N52YF2KY3Z36JYNDGRM2A, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.524553666 (21h9m13.564126786s) [noise=true , address=45.91.116.4:4939 , piece_id=BXLXHF3W5CPAS73RZXC4GL4PPN5PRMVLSSW4ADSW2M2KWRXQY77Q, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.524760001 (21h9m13.564333121s) [noise=true , address=77.90.31.10:4519 , piece_id=5RGPSBJO5NDUDD7Y4NH5CYTGF2FB5RO6ADKEP7YUGZFVMCTACVKA, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.524839723 (21h9m13.564412843s) [noise=true , address=45.91.116.12:4225 , piece_id=JQOKRSTWFE56HRHV4BBEL3ZR7T24XBHVYYU4WA4SWJGF432EPOLQ, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.525132163 (21h9m13.564705293s) [noise=true , address=45.91.116.12:4227 , piece_id=7BZ46KAIEIJDIKHXRLDCMB4Y6E7KSC4QLBOLY2A2NZFA4GVVXD2A, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.525296006 (21h9m13.564869127s) [noise=true , address=45.91.116.6:4975 , piece_id=7U2D53SCSJ5GUGLFWEH6FA66Q5U7EIK7YVBTP5GP5P5OOAVOHQWA, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.525386890 (21h9m13.56496001s) [noise=true , address=45.91.116.10:4083 , piece_id=2XGLF7IS7LC7TU5X2M27UEYQBVX3UML3UEWSFUISJOXF6AVKITTQ, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.525611531 (21h9m13.565184651s) [noise=true , address=45.91.116.7:4009 , piece_id=TYKXIJX7XIMFYKEA5RMORUFVO7UDVKNVBSJAIFUEIB3A76G4HN3Q, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.525648361 (21h9m13.565221481s) [noise=true , address=45.91.116.14:4191 , piece_id=BLZ4EZD5DQ7QQNNRLNCKAB2CQBZGQ2SYKGJJIOZCXNGS4SYOFNXA, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.525870506 (21h9m13.565443626s) [noise=true , address=45.91.116.11:4117 , piece_id=5E2JPHN5T2SETJNOGGAESLRU7PRPUCUDEO25QDW6PA73EKVGMRYQ, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.525935922 (21h9m13.565509042s) [noise=true , address=45.91.116.7:4010 , piece_id=UQRAOTS6AILKBAR6QC3ZD3VATJEYM2T6T4K2KDVU3XRKYQSEB66A, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.526222790 (21h9m13.56579591s) [noise=true , address=45.91.116.13:4153 , piece_id=KUSBCYGRA2XFXZPOYJG4EYGZ7FOSKPKV2EEFQHUND6TPTNQX7AGQ, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.526424746 (21h9m13.565997866s) [noise=true , address=77.90.31.10:4518 , piece_id=XBJTZKIXGIW3WPKU4RWYRZBMAC67YYDYRNAJNBWRYX6SHU3T73WA, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:39.362042598 (21h9m15.401615708s) [upload=18635]: Next returned nil segment Sep 7 10:37:39.362059750 (21h9m15.40163287s) [upload=18635]: Waiting for error group managing segments... Sep 7 10:37:39.669685089 (21h9m15.709258209s) [noise=true , address=45.91.116.14:4191 , piece_id=BLZ4EZD5DQ7QQNNRLNCKAB2CQBZGQ2SYKGJJIOZCXNGS4SYOFNXA, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.679437981 (21h9m15.719011101s) [noise=true , address=77.90.31.10:4519 , piece_id=5RGPSBJO5NDUDD7Y4NH5CYTGF2FB5RO6ADKEP7YUGZFVMCTACVKA, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.679555526 (21h9m15.719128646s) [noise=true , address=45.91.116.12:4225 , piece_id=JQOKRSTWFE56HRHV4BBEL3ZR7T24XBHVYYU4WA4SWJGF432EPOLQ, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.687220604 (21h9m15.726793724s) [noise=true , address=45.91.116.7:4010 , piece_id=UQRAOTS6AILKBAR6QC3ZD3VATJEYM2T6T4K2KDVU3XRKYQSEB66A, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.711092863 (21h9m15.750665983s) [noise=true , address=45.91.116.4:4939 , piece_id=BXLXHF3W5CPAS73RZXC4GL4PPN5PRMVLSSW4ADSW2M2KWRXQY77Q, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.734404882 (21h9m15.773978002s) [noise=true , address=45.91.116.7:4009 , piece_id=TYKXIJX7XIMFYKEA5RMORUFVO7UDVKNVBSJAIFUEIB3A76G4HN3Q, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.743143673 (21h9m15.782716793s) [noise=true , address=45.91.116.11:4119 , piece_id=Y5TR5QSARCJ3HXQZN6OWO7NTLMSGCSLCOEZVJRSB3VZQQL7O57NA, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.748574906 (21h9m15.788148016s) [noise=true , address=45.91.116.6:4974 , piece_id=ZUB4DD6ZXILUSYESNMGNOCKZVTPUY24N52YF2KY3Z36JYNDGRM2A, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.754311293 (21h9m15.793884413s) [noise=true , address=45.91.116.13:4153 , piece_id=KUSBCYGRA2XFXZPOYJG4EYGZ7FOSKPKV2EEFQHUND6TPTNQX7AGQ, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.755384276 (21h9m15.794957396s) [noise=true , address=45.91.116.5:4298 , piece_id=T4KK43URSFHH2LDPBX4CN66G4KVBCWV7JSJM3CADPG6EYGOS2HFQ, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.758010662 (21h9m15.797583782s) [noise=true , address=77.90.31.10:4518 , piece_id=XBJTZKIXGIW3WPKU4RWYRZBMAC67YYDYRNAJNBWRYX6SHU3T73WA, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.774473184 (21h9m15.814046304s) [noise=true , address=45.91.116.6:4973 , piece_id=X63L2GPVONW7ORT5DYOQIHV46EG4DIHEB6TLOZMXKLCQPR5EWDFA, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.817437797 (21h9m15.857010917s) [noise=true , address=45.91.116.11:4117 , piece_id=5E2JPHN5T2SETJNOGGAESLRU7PRPUCUDEO25QDW6PA73EKVGMRYQ, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.817462975 (21h9m15.857036096s) [seg_pos={0 0}, upload=18635]: Segment reached optimal threshold of 13 pieces. Sep 7 10:37:39.817597944 (21h9m15.857171064s) [noise=true , address=45.91.116.6:4975 , piece_id=7U2D53SCSJ5GUGLFWEH6FA66Q5U7EIK7YVBTP5GP5P5OOAVOHQWA, seg_pos={0 0}, upload=18635]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: expected piece hash; context canceled; EOF Sep 7 10:37:39.817683578 (21h9m15.857256698s) [noise=true , address=45.91.116.10:4083 , piece_id=2XGLF7IS7LC7TU5X2M27UEYQBVX3UML3UEWSFUISJOXF6AVKITTQ, seg_pos={0 0}, upload=18635]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: expected piece hash; context canceled; EOF Sep 7 10:37:39.817775434 (21h9m15.857348554s) [noise=true , address=45.91.116.12:4227 , piece_id=7BZ46KAIEIJDIKHXRLDCMB4Y6E7KSC4QLBOLY2A2NZFA4GVVXD2A, seg_pos={0 0}, upload=18635]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH): protocol: expected piece hash; context canceled; EOF Sep 7 10:37:39.817852270 (21h9m15.8574254s) [seg_pos={0 0}, upload=18635]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 7 10:37:39.817955087 (21h9m15.857528217s) [upload=18635]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 7 10:37:39.817961028 (21h9m15.857534148s) [upload=18635]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 7 10:37:40.298833319 (21h9m16.338406429s) [upload=18635]: Done uploading segments... Sep 7 10:37:40.298849160 (21h9m16.33842228s) [upload=18635]: Upload finished. err: <nil> Duration: 6.26357584s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [X.................................................X.........] duration: 9.295712505s FAIL (2025-09-07 10:44:12 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Sep 7 10:37:37.259810434 (21h9m13.299383554s) [upload=18635]: Starting upload Sep 7 10:37:37.259869848 (21h9m13.299442968s) [upload=18635]: Done starting upload Sep 7 10:37:37.259991862 (21h9m13.299564982s) [upload=18635]: Uploading segments... Sep 7 10:37:37.260091512 (21h9m13.299664632s) [upload=18635]: Got next segment. Inline: false Sep 7 10:37:37.260124284 (21h9m13.299697405s) [upload=18635]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 7 10:37:37.260129676 (21h9m13.299702796s) [upload=18635]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 7 10:37:37.523765418 (21h9m13.563338538s) [seg_pos={0 0}, upload=18635]: Begin upload segment... Sep 7 10:37:37.523848977 (21h9m13.563422098s) [seg_pos={0 0}, upload=18635]: Done begin upload segment. Sep 7 10:37:37.523875308 (21h9m13.563448418s) [noise=true , address=45.91.116.11:4119 , piece_id=Y5TR5QSARCJ3HXQZN6OWO7NTLMSGCSLCOEZVJRSB3VZQQL7O57NA, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.523927959 (21h9m13.563501079s) [noise=true , address=45.91.116.5:4298 , piece_id=T4KK43URSFHH2LDPBX4CN66G4KVBCWV7JSJM3CADPG6EYGOS2HFQ, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.524262138 (21h9m13.563835258s) [noise=true , address=45.91.116.6:4973 , piece_id=X63L2GPVONW7ORT5DYOQIHV46EG4DIHEB6TLOZMXKLCQPR5EWDFA, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.524469726 (21h9m13.564042846s) [noise=true , address=45.91.116.6:4974 , piece_id=ZUB4DD6ZXILUSYESNMGNOCKZVTPUY24N52YF2KY3Z36JYNDGRM2A, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.524553666 (21h9m13.564126786s) [noise=true , address=45.91.116.4:4939 , piece_id=BXLXHF3W5CPAS73RZXC4GL4PPN5PRMVLSSW4ADSW2M2KWRXQY77Q, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.524760001 (21h9m13.564333121s) [noise=true , address=77.90.31.10:4519 , piece_id=5RGPSBJO5NDUDD7Y4NH5CYTGF2FB5RO6ADKEP7YUGZFVMCTACVKA, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.524839723 (21h9m13.564412843s) [noise=true , address=45.91.116.12:4225 , piece_id=JQOKRSTWFE56HRHV4BBEL3ZR7T24XBHVYYU4WA4SWJGF432EPOLQ, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.525132163 (21h9m13.564705293s) [noise=true , address=45.91.116.12:4227 , piece_id=7BZ46KAIEIJDIKHXRLDCMB4Y6E7KSC4QLBOLY2A2NZFA4GVVXD2A, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.525296006 (21h9m13.564869127s) [noise=true , address=45.91.116.6:4975 , piece_id=7U2D53SCSJ5GUGLFWEH6FA66Q5U7EIK7YVBTP5GP5P5OOAVOHQWA, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.525386890 (21h9m13.56496001s) [noise=true , address=45.91.116.10:4083 , piece_id=2XGLF7IS7LC7TU5X2M27UEYQBVX3UML3UEWSFUISJOXF6AVKITTQ, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.525611531 (21h9m13.565184651s) [noise=true , address=45.91.116.7:4009 , piece_id=TYKXIJX7XIMFYKEA5RMORUFVO7UDVKNVBSJAIFUEIB3A76G4HN3Q, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.525648361 (21h9m13.565221481s) [noise=true , address=45.91.116.14:4191 , piece_id=BLZ4EZD5DQ7QQNNRLNCKAB2CQBZGQ2SYKGJJIOZCXNGS4SYOFNXA, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.525870506 (21h9m13.565443626s) [noise=true , address=45.91.116.11:4117 , piece_id=5E2JPHN5T2SETJNOGGAESLRU7PRPUCUDEO25QDW6PA73EKVGMRYQ, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.525935922 (21h9m13.565509042s) [noise=true , address=45.91.116.7:4010 , piece_id=UQRAOTS6AILKBAR6QC3ZD3VATJEYM2T6T4K2KDVU3XRKYQSEB66A, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.526222790 (21h9m13.56579591s) [noise=true , address=45.91.116.13:4153 , piece_id=KUSBCYGRA2XFXZPOYJG4EYGZ7FOSKPKV2EEFQHUND6TPTNQX7AGQ, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.526424746 (21h9m13.565997866s) [noise=true , address=77.90.31.10:4518 , piece_id=XBJTZKIXGIW3WPKU4RWYRZBMAC67YYDYRNAJNBWRYX6SHU3T73WA, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:39.362042598 (21h9m15.401615708s) [upload=18635]: Next returned nil segment Sep 7 10:37:39.362059750 (21h9m15.40163287s) [upload=18635]: Waiting for error group managing segments... Sep 7 10:37:39.669685089 (21h9m15.709258209s) [noise=true , address=45.91.116.14:4191 , piece_id=BLZ4EZD5DQ7QQNNRLNCKAB2CQBZGQ2SYKGJJIOZCXNGS4SYOFNXA, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.679437981 (21h9m15.719011101s) [noise=true , address=77.90.31.10:4519 , piece_id=5RGPSBJO5NDUDD7Y4NH5CYTGF2FB5RO6ADKEP7YUGZFVMCTACVKA, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.679555526 (21h9m15.719128646s) [noise=true , address=45.91.116.12:4225 , piece_id=JQOKRSTWFE56HRHV4BBEL3ZR7T24XBHVYYU4WA4SWJGF432EPOLQ, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.687220604 (21h9m15.726793724s) [noise=true , address=45.91.116.7:4010 , piece_id=UQRAOTS6AILKBAR6QC3ZD3VATJEYM2T6T4K2KDVU3XRKYQSEB66A, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.711092863 (21h9m15.750665983s) [noise=true , address=45.91.116.4:4939 , piece_id=BXLXHF3W5CPAS73RZXC4GL4PPN5PRMVLSSW4ADSW2M2KWRXQY77Q, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.734404882 (21h9m15.773978002s) [noise=true , address=45.91.116.7:4009 , piece_id=TYKXIJX7XIMFYKEA5RMORUFVO7UDVKNVBSJAIFUEIB3A76G4HN3Q, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.743143673 (21h9m15.782716793s) [noise=true , address=45.91.116.11:4119 , piece_id=Y5TR5QSARCJ3HXQZN6OWO7NTLMSGCSLCOEZVJRSB3VZQQL7O57NA, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.748574906 (21h9m15.788148016s) [noise=true , address=45.91.116.6:4974 , piece_id=ZUB4DD6ZXILUSYESNMGNOCKZVTPUY24N52YF2KY3Z36JYNDGRM2A, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.754311293 (21h9m15.793884413s) [noise=true , address=45.91.116.13:4153 , piece_id=KUSBCYGRA2XFXZPOYJG4EYGZ7FOSKPKV2EEFQHUND6TPTNQX7AGQ, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.755384276 (21h9m15.794957396s) [noise=true , address=45.91.116.5:4298 , piece_id=T4KK43URSFHH2LDPBX4CN66G4KVBCWV7JSJM3CADPG6EYGOS2HFQ, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.758010662 (21h9m15.797583782s) [noise=true , address=77.90.31.10:4518 , piece_id=XBJTZKIXGIW3WPKU4RWYRZBMAC67YYDYRNAJNBWRYX6SHU3T73WA, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.774473184 (21h9m15.814046304s) [noise=true , address=45.91.116.6:4973 , piece_id=X63L2GPVONW7ORT5DYOQIHV46EG4DIHEB6TLOZMXKLCQPR5EWDFA, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.817437797 (21h9m15.857010917s) [noise=true , address=45.91.116.11:4117 , piece_id=5E2JPHN5T2SETJNOGGAESLRU7PRPUCUDEO25QDW6PA73EKVGMRYQ, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.817462975 (21h9m15.857036096s) [seg_pos={0 0}, upload=18635]: Segment reached optimal threshold of 13 pieces. Sep 7 10:37:39.817597944 (21h9m15.857171064s) [noise=true , address=45.91.116.6:4975 , piece_id=7U2D53SCSJ5GUGLFWEH6FA66Q5U7EIK7YVBTP5GP5P5OOAVOHQWA, seg_pos={0 0}, upload=18635]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: expected piece hash; context canceled; EOF Sep 7 10:37:39.817683578 (21h9m15.857256698s) [noise=true , address=45.91.116.10:4083 , piece_id=2XGLF7IS7LC7TU5X2M27UEYQBVX3UML3UEWSFUISJOXF6AVKITTQ, seg_pos={0 0}, upload=18635]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: expected piece hash; context canceled; EOF Sep 7 10:37:39.817775434 (21h9m15.857348554s) [noise=true , address=45.91.116.12:4227 , piece_id=7BZ46KAIEIJDIKHXRLDCMB4Y6E7KSC4QLBOLY2A2NZFA4GVVXD2A, seg_pos={0 0}, upload=18635]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH): protocol: expected piece hash; context canceled; EOF Sep 7 10:37:39.817852270 (21h9m15.8574254s) [seg_pos={0 0}, upload=18635]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 7 10:37:39.817955087 (21h9m15.857528217s) [upload=18635]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 7 10:37:39.817961028 (21h9m15.857534148s) [upload=18635]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 7 10:37:40.298833319 (21h9m16.338406429s) [upload=18635]: Done uploading segments... Sep 7 10:37:40.298849160 (21h9m16.33842228s) [upload=18635]: Upload finished. err: <nil> Duration: 9.295740057s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.X.................................................X........] duration: 24.402605147s FAIL (2025-09-07 10:43:47 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Sep 7 10:37:37.259810434 (21h9m13.299383554s) [upload=18635]: Starting upload Sep 7 10:37:37.259869848 (21h9m13.299442968s) [upload=18635]: Done starting upload Sep 7 10:37:37.259991862 (21h9m13.299564982s) [upload=18635]: Uploading segments... Sep 7 10:37:37.260091512 (21h9m13.299664632s) [upload=18635]: Got next segment. Inline: false Sep 7 10:37:37.260124284 (21h9m13.299697405s) [upload=18635]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 7 10:37:37.260129676 (21h9m13.299702796s) [upload=18635]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 7 10:37:37.523765418 (21h9m13.563338538s) [seg_pos={0 0}, upload=18635]: Begin upload segment... Sep 7 10:37:37.523848977 (21h9m13.563422098s) [seg_pos={0 0}, upload=18635]: Done begin upload segment. Sep 7 10:37:37.523875308 (21h9m13.563448418s) [noise=true , address=45.91.116.11:4119 , piece_id=Y5TR5QSARCJ3HXQZN6OWO7NTLMSGCSLCOEZVJRSB3VZQQL7O57NA, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.523927959 (21h9m13.563501079s) [noise=true , address=45.91.116.5:4298 , piece_id=T4KK43URSFHH2LDPBX4CN66G4KVBCWV7JSJM3CADPG6EYGOS2HFQ, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.524262138 (21h9m13.563835258s) [noise=true , address=45.91.116.6:4973 , piece_id=X63L2GPVONW7ORT5DYOQIHV46EG4DIHEB6TLOZMXKLCQPR5EWDFA, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.524469726 (21h9m13.564042846s) [noise=true , address=45.91.116.6:4974 , piece_id=ZUB4DD6ZXILUSYESNMGNOCKZVTPUY24N52YF2KY3Z36JYNDGRM2A, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.524553666 (21h9m13.564126786s) [noise=true , address=45.91.116.4:4939 , piece_id=BXLXHF3W5CPAS73RZXC4GL4PPN5PRMVLSSW4ADSW2M2KWRXQY77Q, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.524760001 (21h9m13.564333121s) [noise=true , address=77.90.31.10:4519 , piece_id=5RGPSBJO5NDUDD7Y4NH5CYTGF2FB5RO6ADKEP7YUGZFVMCTACVKA, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.524839723 (21h9m13.564412843s) [noise=true , address=45.91.116.12:4225 , piece_id=JQOKRSTWFE56HRHV4BBEL3ZR7T24XBHVYYU4WA4SWJGF432EPOLQ, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.525132163 (21h9m13.564705293s) [noise=true , address=45.91.116.12:4227 , piece_id=7BZ46KAIEIJDIKHXRLDCMB4Y6E7KSC4QLBOLY2A2NZFA4GVVXD2A, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.525296006 (21h9m13.564869127s) [noise=true , address=45.91.116.6:4975 , piece_id=7U2D53SCSJ5GUGLFWEH6FA66Q5U7EIK7YVBTP5GP5P5OOAVOHQWA, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.525386890 (21h9m13.56496001s) [noise=true , address=45.91.116.10:4083 , piece_id=2XGLF7IS7LC7TU5X2M27UEYQBVX3UML3UEWSFUISJOXF6AVKITTQ, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.525611531 (21h9m13.565184651s) [noise=true , address=45.91.116.7:4009 , piece_id=TYKXIJX7XIMFYKEA5RMORUFVO7UDVKNVBSJAIFUEIB3A76G4HN3Q, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.525648361 (21h9m13.565221481s) [noise=true , address=45.91.116.14:4191 , piece_id=BLZ4EZD5DQ7QQNNRLNCKAB2CQBZGQ2SYKGJJIOZCXNGS4SYOFNXA, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.525870506 (21h9m13.565443626s) [noise=true , address=45.91.116.11:4117 , piece_id=5E2JPHN5T2SETJNOGGAESLRU7PRPUCUDEO25QDW6PA73EKVGMRYQ, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.525935922 (21h9m13.565509042s) [noise=true , address=45.91.116.7:4010 , piece_id=UQRAOTS6AILKBAR6QC3ZD3VATJEYM2T6T4K2KDVU3XRKYQSEB66A, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.526222790 (21h9m13.56579591s) [noise=true , address=45.91.116.13:4153 , piece_id=KUSBCYGRA2XFXZPOYJG4EYGZ7FOSKPKV2EEFQHUND6TPTNQX7AGQ, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.526424746 (21h9m13.565997866s) [noise=true , address=77.90.31.10:4518 , piece_id=XBJTZKIXGIW3WPKU4RWYRZBMAC67YYDYRNAJNBWRYX6SHU3T73WA, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:39.362042598 (21h9m15.401615708s) [upload=18635]: Next returned nil segment Sep 7 10:37:39.362059750 (21h9m15.40163287s) [upload=18635]: Waiting for error group managing segments... Sep 7 10:37:39.669685089 (21h9m15.709258209s) [noise=true , address=45.91.116.14:4191 , piece_id=BLZ4EZD5DQ7QQNNRLNCKAB2CQBZGQ2SYKGJJIOZCXNGS4SYOFNXA, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.679437981 (21h9m15.719011101s) [noise=true , address=77.90.31.10:4519 , piece_id=5RGPSBJO5NDUDD7Y4NH5CYTGF2FB5RO6ADKEP7YUGZFVMCTACVKA, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.679555526 (21h9m15.719128646s) [noise=true , address=45.91.116.12:4225 , piece_id=JQOKRSTWFE56HRHV4BBEL3ZR7T24XBHVYYU4WA4SWJGF432EPOLQ, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.687220604 (21h9m15.726793724s) [noise=true , address=45.91.116.7:4010 , piece_id=UQRAOTS6AILKBAR6QC3ZD3VATJEYM2T6T4K2KDVU3XRKYQSEB66A, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.711092863 (21h9m15.750665983s) [noise=true , address=45.91.116.4:4939 , piece_id=BXLXHF3W5CPAS73RZXC4GL4PPN5PRMVLSSW4ADSW2M2KWRXQY77Q, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.734404882 (21h9m15.773978002s) [noise=true , address=45.91.116.7:4009 , piece_id=TYKXIJX7XIMFYKEA5RMORUFVO7UDVKNVBSJAIFUEIB3A76G4HN3Q, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.743143673 (21h9m15.782716793s) [noise=true , address=45.91.116.11:4119 , piece_id=Y5TR5QSARCJ3HXQZN6OWO7NTLMSGCSLCOEZVJRSB3VZQQL7O57NA, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.748574906 (21h9m15.788148016s) [noise=true , address=45.91.116.6:4974 , piece_id=ZUB4DD6ZXILUSYESNMGNOCKZVTPUY24N52YF2KY3Z36JYNDGRM2A, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.754311293 (21h9m15.793884413s) [noise=true , address=45.91.116.13:4153 , piece_id=KUSBCYGRA2XFXZPOYJG4EYGZ7FOSKPKV2EEFQHUND6TPTNQX7AGQ, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.755384276 (21h9m15.794957396s) [noise=true , address=45.91.116.5:4298 , piece_id=T4KK43URSFHH2LDPBX4CN66G4KVBCWV7JSJM3CADPG6EYGOS2HFQ, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.758010662 (21h9m15.797583782s) [noise=true , address=77.90.31.10:4518 , piece_id=XBJTZKIXGIW3WPKU4RWYRZBMAC67YYDYRNAJNBWRYX6SHU3T73WA, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.774473184 (21h9m15.814046304s) [noise=true , address=45.91.116.6:4973 , piece_id=X63L2GPVONW7ORT5DYOQIHV46EG4DIHEB6TLOZMXKLCQPR5EWDFA, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.817437797 (21h9m15.857010917s) [noise=true , address=45.91.116.11:4117 , piece_id=5E2JPHN5T2SETJNOGGAESLRU7PRPUCUDEO25QDW6PA73EKVGMRYQ, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.817462975 (21h9m15.857036096s) [seg_pos={0 0}, upload=18635]: Segment reached optimal threshold of 13 pieces. Sep 7 10:37:39.817597944 (21h9m15.857171064s) [noise=true , address=45.91.116.6:4975 , piece_id=7U2D53SCSJ5GUGLFWEH6FA66Q5U7EIK7YVBTP5GP5P5OOAVOHQWA, seg_pos={0 0}, upload=18635]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: expected piece hash; context canceled; EOF Sep 7 10:37:39.817683578 (21h9m15.857256698s) [noise=true , address=45.91.116.10:4083 , piece_id=2XGLF7IS7LC7TU5X2M27UEYQBVX3UML3UEWSFUISJOXF6AVKITTQ, seg_pos={0 0}, upload=18635]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: expected piece hash; context canceled; EOF Sep 7 10:37:39.817775434 (21h9m15.857348554s) [noise=true , address=45.91.116.12:4227 , piece_id=7BZ46KAIEIJDIKHXRLDCMB4Y6E7KSC4QLBOLY2A2NZFA4GVVXD2A, seg_pos={0 0}, upload=18635]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH): protocol: expected piece hash; context canceled; EOF Sep 7 10:37:39.817852270 (21h9m15.8574254s) [seg_pos={0 0}, upload=18635]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 7 10:37:39.817955087 (21h9m15.857528217s) [upload=18635]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 7 10:37:39.817961028 (21h9m15.857534148s) [upload=18635]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 7 10:37:40.298833319 (21h9m16.338406429s) [upload=18635]: Done uploading segments... Sep 7 10:37:40.298849160 (21h9m16.33842228s) [upload=18635]: Upload finished. err: <nil> Duration: 24.402629584s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..X.................................................X.......] duration: 9.301609438s FAIL (2025-09-07 10:43:16 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Sep 7 10:37:37.259810434 (21h9m13.299383554s) [upload=18635]: Starting upload Sep 7 10:37:37.259869848 (21h9m13.299442968s) [upload=18635]: Done starting upload Sep 7 10:37:37.259991862 (21h9m13.299564982s) [upload=18635]: Uploading segments... Sep 7 10:37:37.260091512 (21h9m13.299664632s) [upload=18635]: Got next segment. Inline: false Sep 7 10:37:37.260124284 (21h9m13.299697405s) [upload=18635]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 7 10:37:37.260129676 (21h9m13.299702796s) [upload=18635]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 7 10:37:37.523765418 (21h9m13.563338538s) [seg_pos={0 0}, upload=18635]: Begin upload segment... Sep 7 10:37:37.523848977 (21h9m13.563422098s) [seg_pos={0 0}, upload=18635]: Done begin upload segment. Sep 7 10:37:37.523875308 (21h9m13.563448418s) [noise=true , address=45.91.116.11:4119 , piece_id=Y5TR5QSARCJ3HXQZN6OWO7NTLMSGCSLCOEZVJRSB3VZQQL7O57NA, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.523927959 (21h9m13.563501079s) [noise=true , address=45.91.116.5:4298 , piece_id=T4KK43URSFHH2LDPBX4CN66G4KVBCWV7JSJM3CADPG6EYGOS2HFQ, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.524262138 (21h9m13.563835258s) [noise=true , address=45.91.116.6:4973 , piece_id=X63L2GPVONW7ORT5DYOQIHV46EG4DIHEB6TLOZMXKLCQPR5EWDFA, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.524469726 (21h9m13.564042846s) [noise=true , address=45.91.116.6:4974 , piece_id=ZUB4DD6ZXILUSYESNMGNOCKZVTPUY24N52YF2KY3Z36JYNDGRM2A, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.524553666 (21h9m13.564126786s) [noise=true , address=45.91.116.4:4939 , piece_id=BXLXHF3W5CPAS73RZXC4GL4PPN5PRMVLSSW4ADSW2M2KWRXQY77Q, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.524760001 (21h9m13.564333121s) [noise=true , address=77.90.31.10:4519 , piece_id=5RGPSBJO5NDUDD7Y4NH5CYTGF2FB5RO6ADKEP7YUGZFVMCTACVKA, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.524839723 (21h9m13.564412843s) [noise=true , address=45.91.116.12:4225 , piece_id=JQOKRSTWFE56HRHV4BBEL3ZR7T24XBHVYYU4WA4SWJGF432EPOLQ, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.525132163 (21h9m13.564705293s) [noise=true , address=45.91.116.12:4227 , piece_id=7BZ46KAIEIJDIKHXRLDCMB4Y6E7KSC4QLBOLY2A2NZFA4GVVXD2A, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.525296006 (21h9m13.564869127s) [noise=true , address=45.91.116.6:4975 , piece_id=7U2D53SCSJ5GUGLFWEH6FA66Q5U7EIK7YVBTP5GP5P5OOAVOHQWA, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.525386890 (21h9m13.56496001s) [noise=true , address=45.91.116.10:4083 , piece_id=2XGLF7IS7LC7TU5X2M27UEYQBVX3UML3UEWSFUISJOXF6AVKITTQ, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.525611531 (21h9m13.565184651s) [noise=true , address=45.91.116.7:4009 , piece_id=TYKXIJX7XIMFYKEA5RMORUFVO7UDVKNVBSJAIFUEIB3A76G4HN3Q, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.525648361 (21h9m13.565221481s) [noise=true , address=45.91.116.14:4191 , piece_id=BLZ4EZD5DQ7QQNNRLNCKAB2CQBZGQ2SYKGJJIOZCXNGS4SYOFNXA, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.525870506 (21h9m13.565443626s) [noise=true , address=45.91.116.11:4117 , piece_id=5E2JPHN5T2SETJNOGGAESLRU7PRPUCUDEO25QDW6PA73EKVGMRYQ, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.525935922 (21h9m13.565509042s) [noise=true , address=45.91.116.7:4010 , piece_id=UQRAOTS6AILKBAR6QC3ZD3VATJEYM2T6T4K2KDVU3XRKYQSEB66A, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.526222790 (21h9m13.56579591s) [noise=true , address=45.91.116.13:4153 , piece_id=KUSBCYGRA2XFXZPOYJG4EYGZ7FOSKPKV2EEFQHUND6TPTNQX7AGQ, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.526424746 (21h9m13.565997866s) [noise=true , address=77.90.31.10:4518 , piece_id=XBJTZKIXGIW3WPKU4RWYRZBMAC67YYDYRNAJNBWRYX6SHU3T73WA, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:39.362042598 (21h9m15.401615708s) [upload=18635]: Next returned nil segment Sep 7 10:37:39.362059750 (21h9m15.40163287s) [upload=18635]: Waiting for error group managing segments... Sep 7 10:37:39.669685089 (21h9m15.709258209s) [noise=true , address=45.91.116.14:4191 , piece_id=BLZ4EZD5DQ7QQNNRLNCKAB2CQBZGQ2SYKGJJIOZCXNGS4SYOFNXA, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.679437981 (21h9m15.719011101s) [noise=true , address=77.90.31.10:4519 , piece_id=5RGPSBJO5NDUDD7Y4NH5CYTGF2FB5RO6ADKEP7YUGZFVMCTACVKA, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.679555526 (21h9m15.719128646s) [noise=true , address=45.91.116.12:4225 , piece_id=JQOKRSTWFE56HRHV4BBEL3ZR7T24XBHVYYU4WA4SWJGF432EPOLQ, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.687220604 (21h9m15.726793724s) [noise=true , address=45.91.116.7:4010 , piece_id=UQRAOTS6AILKBAR6QC3ZD3VATJEYM2T6T4K2KDVU3XRKYQSEB66A, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.711092863 (21h9m15.750665983s) [noise=true , address=45.91.116.4:4939 , piece_id=BXLXHF3W5CPAS73RZXC4GL4PPN5PRMVLSSW4ADSW2M2KWRXQY77Q, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.734404882 (21h9m15.773978002s) [noise=true , address=45.91.116.7:4009 , piece_id=TYKXIJX7XIMFYKEA5RMORUFVO7UDVKNVBSJAIFUEIB3A76G4HN3Q, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.743143673 (21h9m15.782716793s) [noise=true , address=45.91.116.11:4119 , piece_id=Y5TR5QSARCJ3HXQZN6OWO7NTLMSGCSLCOEZVJRSB3VZQQL7O57NA, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.748574906 (21h9m15.788148016s) [noise=true , address=45.91.116.6:4974 , piece_id=ZUB4DD6ZXILUSYESNMGNOCKZVTPUY24N52YF2KY3Z36JYNDGRM2A, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.754311293 (21h9m15.793884413s) [noise=true , address=45.91.116.13:4153 , piece_id=KUSBCYGRA2XFXZPOYJG4EYGZ7FOSKPKV2EEFQHUND6TPTNQX7AGQ, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.755384276 (21h9m15.794957396s) [noise=true , address=45.91.116.5:4298 , piece_id=T4KK43URSFHH2LDPBX4CN66G4KVBCWV7JSJM3CADPG6EYGOS2HFQ, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.758010662 (21h9m15.797583782s) [noise=true , address=77.90.31.10:4518 , piece_id=XBJTZKIXGIW3WPKU4RWYRZBMAC67YYDYRNAJNBWRYX6SHU3T73WA, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.774473184 (21h9m15.814046304s) [noise=true , address=45.91.116.6:4973 , piece_id=X63L2GPVONW7ORT5DYOQIHV46EG4DIHEB6TLOZMXKLCQPR5EWDFA, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.817437797 (21h9m15.857010917s) [noise=true , address=45.91.116.11:4117 , piece_id=5E2JPHN5T2SETJNOGGAESLRU7PRPUCUDEO25QDW6PA73EKVGMRYQ, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.817462975 (21h9m15.857036096s) [seg_pos={0 0}, upload=18635]: Segment reached optimal threshold of 13 pieces. Sep 7 10:37:39.817597944 (21h9m15.857171064s) [noise=true , address=45.91.116.6:4975 , piece_id=7U2D53SCSJ5GUGLFWEH6FA66Q5U7EIK7YVBTP5GP5P5OOAVOHQWA, seg_pos={0 0}, upload=18635]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: expected piece hash; context canceled; EOF Sep 7 10:37:39.817683578 (21h9m15.857256698s) [noise=true , address=45.91.116.10:4083 , piece_id=2XGLF7IS7LC7TU5X2M27UEYQBVX3UML3UEWSFUISJOXF6AVKITTQ, seg_pos={0 0}, upload=18635]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: expected piece hash; context canceled; EOF Sep 7 10:37:39.817775434 (21h9m15.857348554s) [noise=true , address=45.91.116.12:4227 , piece_id=7BZ46KAIEIJDIKHXRLDCMB4Y6E7KSC4QLBOLY2A2NZFA4GVVXD2A, seg_pos={0 0}, upload=18635]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH): protocol: expected piece hash; context canceled; EOF Sep 7 10:37:39.817852270 (21h9m15.8574254s) [seg_pos={0 0}, upload=18635]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 7 10:37:39.817955087 (21h9m15.857528217s) [upload=18635]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 7 10:37:39.817961028 (21h9m15.857534148s) [upload=18635]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 7 10:37:40.298833319 (21h9m16.338406429s) [upload=18635]: Done uploading segments... Sep 7 10:37:40.298849160 (21h9m16.33842228s) [upload=18635]: Upload finished. err: <nil> Duration: 9.301640237s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...X.................................................X......] duration: 6.039899148s FAIL (2025-09-07 10:42:11 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Sep 7 10:37:37.259810434 (21h9m13.299383554s) [upload=18635]: Starting upload Sep 7 10:37:37.259869848 (21h9m13.299442968s) [upload=18635]: Done starting upload Sep 7 10:37:37.259991862 (21h9m13.299564982s) [upload=18635]: Uploading segments... Sep 7 10:37:37.260091512 (21h9m13.299664632s) [upload=18635]: Got next segment. Inline: false Sep 7 10:37:37.260124284 (21h9m13.299697405s) [upload=18635]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 7 10:37:37.260129676 (21h9m13.299702796s) [upload=18635]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 7 10:37:37.523765418 (21h9m13.563338538s) [seg_pos={0 0}, upload=18635]: Begin upload segment... Sep 7 10:37:37.523848977 (21h9m13.563422098s) [seg_pos={0 0}, upload=18635]: Done begin upload segment. Sep 7 10:37:37.523875308 (21h9m13.563448418s) [noise=true , address=45.91.116.11:4119 , piece_id=Y5TR5QSARCJ3HXQZN6OWO7NTLMSGCSLCOEZVJRSB3VZQQL7O57NA, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.523927959 (21h9m13.563501079s) [noise=true , address=45.91.116.5:4298 , piece_id=T4KK43URSFHH2LDPBX4CN66G4KVBCWV7JSJM3CADPG6EYGOS2HFQ, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.524262138 (21h9m13.563835258s) [noise=true , address=45.91.116.6:4973 , piece_id=X63L2GPVONW7ORT5DYOQIHV46EG4DIHEB6TLOZMXKLCQPR5EWDFA, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.524469726 (21h9m13.564042846s) [noise=true , address=45.91.116.6:4974 , piece_id=ZUB4DD6ZXILUSYESNMGNOCKZVTPUY24N52YF2KY3Z36JYNDGRM2A, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.524553666 (21h9m13.564126786s) [noise=true , address=45.91.116.4:4939 , piece_id=BXLXHF3W5CPAS73RZXC4GL4PPN5PRMVLSSW4ADSW2M2KWRXQY77Q, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.524760001 (21h9m13.564333121s) [noise=true , address=77.90.31.10:4519 , piece_id=5RGPSBJO5NDUDD7Y4NH5CYTGF2FB5RO6ADKEP7YUGZFVMCTACVKA, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.524839723 (21h9m13.564412843s) [noise=true , address=45.91.116.12:4225 , piece_id=JQOKRSTWFE56HRHV4BBEL3ZR7T24XBHVYYU4WA4SWJGF432EPOLQ, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.525132163 (21h9m13.564705293s) [noise=true , address=45.91.116.12:4227 , piece_id=7BZ46KAIEIJDIKHXRLDCMB4Y6E7KSC4QLBOLY2A2NZFA4GVVXD2A, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.525296006 (21h9m13.564869127s) [noise=true , address=45.91.116.6:4975 , piece_id=7U2D53SCSJ5GUGLFWEH6FA66Q5U7EIK7YVBTP5GP5P5OOAVOHQWA, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.525386890 (21h9m13.56496001s) [noise=true , address=45.91.116.10:4083 , piece_id=2XGLF7IS7LC7TU5X2M27UEYQBVX3UML3UEWSFUISJOXF6AVKITTQ, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.525611531 (21h9m13.565184651s) [noise=true , address=45.91.116.7:4009 , piece_id=TYKXIJX7XIMFYKEA5RMORUFVO7UDVKNVBSJAIFUEIB3A76G4HN3Q, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.525648361 (21h9m13.565221481s) [noise=true , address=45.91.116.14:4191 , piece_id=BLZ4EZD5DQ7QQNNRLNCKAB2CQBZGQ2SYKGJJIOZCXNGS4SYOFNXA, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.525870506 (21h9m13.565443626s) [noise=true , address=45.91.116.11:4117 , piece_id=5E2JPHN5T2SETJNOGGAESLRU7PRPUCUDEO25QDW6PA73EKVGMRYQ, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.525935922 (21h9m13.565509042s) [noise=true , address=45.91.116.7:4010 , piece_id=UQRAOTS6AILKBAR6QC3ZD3VATJEYM2T6T4K2KDVU3XRKYQSEB66A, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.526222790 (21h9m13.56579591s) [noise=true , address=45.91.116.13:4153 , piece_id=KUSBCYGRA2XFXZPOYJG4EYGZ7FOSKPKV2EEFQHUND6TPTNQX7AGQ, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.526424746 (21h9m13.565997866s) [noise=true , address=77.90.31.10:4518 , piece_id=XBJTZKIXGIW3WPKU4RWYRZBMAC67YYDYRNAJNBWRYX6SHU3T73WA, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:39.362042598 (21h9m15.401615708s) [upload=18635]: Next returned nil segment Sep 7 10:37:39.362059750 (21h9m15.40163287s) [upload=18635]: Waiting for error group managing segments... Sep 7 10:37:39.669685089 (21h9m15.709258209s) [noise=true , address=45.91.116.14:4191 , piece_id=BLZ4EZD5DQ7QQNNRLNCKAB2CQBZGQ2SYKGJJIOZCXNGS4SYOFNXA, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.679437981 (21h9m15.719011101s) [noise=true , address=77.90.31.10:4519 , piece_id=5RGPSBJO5NDUDD7Y4NH5CYTGF2FB5RO6ADKEP7YUGZFVMCTACVKA, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.679555526 (21h9m15.719128646s) [noise=true , address=45.91.116.12:4225 , piece_id=JQOKRSTWFE56HRHV4BBEL3ZR7T24XBHVYYU4WA4SWJGF432EPOLQ, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.687220604 (21h9m15.726793724s) [noise=true , address=45.91.116.7:4010 , piece_id=UQRAOTS6AILKBAR6QC3ZD3VATJEYM2T6T4K2KDVU3XRKYQSEB66A, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.711092863 (21h9m15.750665983s) [noise=true , address=45.91.116.4:4939 , piece_id=BXLXHF3W5CPAS73RZXC4GL4PPN5PRMVLSSW4ADSW2M2KWRXQY77Q, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.734404882 (21h9m15.773978002s) [noise=true , address=45.91.116.7:4009 , piece_id=TYKXIJX7XIMFYKEA5RMORUFVO7UDVKNVBSJAIFUEIB3A76G4HN3Q, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.743143673 (21h9m15.782716793s) [noise=true , address=45.91.116.11:4119 , piece_id=Y5TR5QSARCJ3HXQZN6OWO7NTLMSGCSLCOEZVJRSB3VZQQL7O57NA, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.748574906 (21h9m15.788148016s) [noise=true , address=45.91.116.6:4974 , piece_id=ZUB4DD6ZXILUSYESNMGNOCKZVTPUY24N52YF2KY3Z36JYNDGRM2A, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.754311293 (21h9m15.793884413s) [noise=true , address=45.91.116.13:4153 , piece_id=KUSBCYGRA2XFXZPOYJG4EYGZ7FOSKPKV2EEFQHUND6TPTNQX7AGQ, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.755384276 (21h9m15.794957396s) [noise=true , address=45.91.116.5:4298 , piece_id=T4KK43URSFHH2LDPBX4CN66G4KVBCWV7JSJM3CADPG6EYGOS2HFQ, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.758010662 (21h9m15.797583782s) [noise=true , address=77.90.31.10:4518 , piece_id=XBJTZKIXGIW3WPKU4RWYRZBMAC67YYDYRNAJNBWRYX6SHU3T73WA, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.774473184 (21h9m15.814046304s) [noise=true , address=45.91.116.6:4973 , piece_id=X63L2GPVONW7ORT5DYOQIHV46EG4DIHEB6TLOZMXKLCQPR5EWDFA, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.817437797 (21h9m15.857010917s) [noise=true , address=45.91.116.11:4117 , piece_id=5E2JPHN5T2SETJNOGGAESLRU7PRPUCUDEO25QDW6PA73EKVGMRYQ, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.817462975 (21h9m15.857036096s) [seg_pos={0 0}, upload=18635]: Segment reached optimal threshold of 13 pieces. Sep 7 10:37:39.817597944 (21h9m15.857171064s) [noise=true , address=45.91.116.6:4975 , piece_id=7U2D53SCSJ5GUGLFWEH6FA66Q5U7EIK7YVBTP5GP5P5OOAVOHQWA, seg_pos={0 0}, upload=18635]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: expected piece hash; context canceled; EOF Sep 7 10:37:39.817683578 (21h9m15.857256698s) [noise=true , address=45.91.116.10:4083 , piece_id=2XGLF7IS7LC7TU5X2M27UEYQBVX3UML3UEWSFUISJOXF6AVKITTQ, seg_pos={0 0}, upload=18635]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: expected piece hash; context canceled; EOF Sep 7 10:37:39.817775434 (21h9m15.857348554s) [noise=true , address=45.91.116.12:4227 , piece_id=7BZ46KAIEIJDIKHXRLDCMB4Y6E7KSC4QLBOLY2A2NZFA4GVVXD2A, seg_pos={0 0}, upload=18635]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH): protocol: expected piece hash; context canceled; EOF Sep 7 10:37:39.817852270 (21h9m15.8574254s) [seg_pos={0 0}, upload=18635]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 7 10:37:39.817955087 (21h9m15.857528217s) [upload=18635]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 7 10:37:39.817961028 (21h9m15.857534148s) [upload=18635]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 7 10:37:40.298833319 (21h9m16.338406429s) [upload=18635]: Done uploading segments... Sep 7 10:37:40.298849160 (21h9m16.33842228s) [upload=18635]: Upload finished. err: <nil> Duration: 6.039925158s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [....X.................................................X.....] duration: 17.452220313s FAIL (2025-09-07 10:41:49 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Sep 7 10:37:37.259810434 (21h9m13.299383554s) [upload=18635]: Starting upload Sep 7 10:37:37.259869848 (21h9m13.299442968s) [upload=18635]: Done starting upload Sep 7 10:37:37.259991862 (21h9m13.299564982s) [upload=18635]: Uploading segments... Sep 7 10:37:37.260091512 (21h9m13.299664632s) [upload=18635]: Got next segment. Inline: false Sep 7 10:37:37.260124284 (21h9m13.299697405s) [upload=18635]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 7 10:37:37.260129676 (21h9m13.299702796s) [upload=18635]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 7 10:37:37.523765418 (21h9m13.563338538s) [seg_pos={0 0}, upload=18635]: Begin upload segment... Sep 7 10:37:37.523848977 (21h9m13.563422098s) [seg_pos={0 0}, upload=18635]: Done begin upload segment. Sep 7 10:37:37.523875308 (21h9m13.563448418s) [noise=true , address=45.91.116.11:4119 , piece_id=Y5TR5QSARCJ3HXQZN6OWO7NTLMSGCSLCOEZVJRSB3VZQQL7O57NA, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.523927959 (21h9m13.563501079s) [noise=true , address=45.91.116.5:4298 , piece_id=T4KK43URSFHH2LDPBX4CN66G4KVBCWV7JSJM3CADPG6EYGOS2HFQ, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.524262138 (21h9m13.563835258s) [noise=true , address=45.91.116.6:4973 , piece_id=X63L2GPVONW7ORT5DYOQIHV46EG4DIHEB6TLOZMXKLCQPR5EWDFA, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.524469726 (21h9m13.564042846s) [noise=true , address=45.91.116.6:4974 , piece_id=ZUB4DD6ZXILUSYESNMGNOCKZVTPUY24N52YF2KY3Z36JYNDGRM2A, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.524553666 (21h9m13.564126786s) [noise=true , address=45.91.116.4:4939 , piece_id=BXLXHF3W5CPAS73RZXC4GL4PPN5PRMVLSSW4ADSW2M2KWRXQY77Q, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.524760001 (21h9m13.564333121s) [noise=true , address=77.90.31.10:4519 , piece_id=5RGPSBJO5NDUDD7Y4NH5CYTGF2FB5RO6ADKEP7YUGZFVMCTACVKA, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.524839723 (21h9m13.564412843s) [noise=true , address=45.91.116.12:4225 , piece_id=JQOKRSTWFE56HRHV4BBEL3ZR7T24XBHVYYU4WA4SWJGF432EPOLQ, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.525132163 (21h9m13.564705293s) [noise=true , address=45.91.116.12:4227 , piece_id=7BZ46KAIEIJDIKHXRLDCMB4Y6E7KSC4QLBOLY2A2NZFA4GVVXD2A, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.525296006 (21h9m13.564869127s) [noise=true , address=45.91.116.6:4975 , piece_id=7U2D53SCSJ5GUGLFWEH6FA66Q5U7EIK7YVBTP5GP5P5OOAVOHQWA, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.525386890 (21h9m13.56496001s) [noise=true , address=45.91.116.10:4083 , piece_id=2XGLF7IS7LC7TU5X2M27UEYQBVX3UML3UEWSFUISJOXF6AVKITTQ, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.525611531 (21h9m13.565184651s) [noise=true , address=45.91.116.7:4009 , piece_id=TYKXIJX7XIMFYKEA5RMORUFVO7UDVKNVBSJAIFUEIB3A76G4HN3Q, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.525648361 (21h9m13.565221481s) [noise=true , address=45.91.116.14:4191 , piece_id=BLZ4EZD5DQ7QQNNRLNCKAB2CQBZGQ2SYKGJJIOZCXNGS4SYOFNXA, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.525870506 (21h9m13.565443626s) [noise=true , address=45.91.116.11:4117 , piece_id=5E2JPHN5T2SETJNOGGAESLRU7PRPUCUDEO25QDW6PA73EKVGMRYQ, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.525935922 (21h9m13.565509042s) [noise=true , address=45.91.116.7:4010 , piece_id=UQRAOTS6AILKBAR6QC3ZD3VATJEYM2T6T4K2KDVU3XRKYQSEB66A, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.526222790 (21h9m13.56579591s) [noise=true , address=45.91.116.13:4153 , piece_id=KUSBCYGRA2XFXZPOYJG4EYGZ7FOSKPKV2EEFQHUND6TPTNQX7AGQ, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.526424746 (21h9m13.565997866s) [noise=true , address=77.90.31.10:4518 , piece_id=XBJTZKIXGIW3WPKU4RWYRZBMAC67YYDYRNAJNBWRYX6SHU3T73WA, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:39.362042598 (21h9m15.401615708s) [upload=18635]: Next returned nil segment Sep 7 10:37:39.362059750 (21h9m15.40163287s) [upload=18635]: Waiting for error group managing segments... Sep 7 10:37:39.669685089 (21h9m15.709258209s) [noise=true , address=45.91.116.14:4191 , piece_id=BLZ4EZD5DQ7QQNNRLNCKAB2CQBZGQ2SYKGJJIOZCXNGS4SYOFNXA, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.679437981 (21h9m15.719011101s) [noise=true , address=77.90.31.10:4519 , piece_id=5RGPSBJO5NDUDD7Y4NH5CYTGF2FB5RO6ADKEP7YUGZFVMCTACVKA, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.679555526 (21h9m15.719128646s) [noise=true , address=45.91.116.12:4225 , piece_id=JQOKRSTWFE56HRHV4BBEL3ZR7T24XBHVYYU4WA4SWJGF432EPOLQ, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.687220604 (21h9m15.726793724s) [noise=true , address=45.91.116.7:4010 , piece_id=UQRAOTS6AILKBAR6QC3ZD3VATJEYM2T6T4K2KDVU3XRKYQSEB66A, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.711092863 (21h9m15.750665983s) [noise=true , address=45.91.116.4:4939 , piece_id=BXLXHF3W5CPAS73RZXC4GL4PPN5PRMVLSSW4ADSW2M2KWRXQY77Q, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.734404882 (21h9m15.773978002s) [noise=true , address=45.91.116.7:4009 , piece_id=TYKXIJX7XIMFYKEA5RMORUFVO7UDVKNVBSJAIFUEIB3A76G4HN3Q, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.743143673 (21h9m15.782716793s) [noise=true , address=45.91.116.11:4119 , piece_id=Y5TR5QSARCJ3HXQZN6OWO7NTLMSGCSLCOEZVJRSB3VZQQL7O57NA, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.748574906 (21h9m15.788148016s) [noise=true , address=45.91.116.6:4974 , piece_id=ZUB4DD6ZXILUSYESNMGNOCKZVTPUY24N52YF2KY3Z36JYNDGRM2A, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.754311293 (21h9m15.793884413s) [noise=true , address=45.91.116.13:4153 , piece_id=KUSBCYGRA2XFXZPOYJG4EYGZ7FOSKPKV2EEFQHUND6TPTNQX7AGQ, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.755384276 (21h9m15.794957396s) [noise=true , address=45.91.116.5:4298 , piece_id=T4KK43URSFHH2LDPBX4CN66G4KVBCWV7JSJM3CADPG6EYGOS2HFQ, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.758010662 (21h9m15.797583782s) [noise=true , address=77.90.31.10:4518 , piece_id=XBJTZKIXGIW3WPKU4RWYRZBMAC67YYDYRNAJNBWRYX6SHU3T73WA, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.774473184 (21h9m15.814046304s) [noise=true , address=45.91.116.6:4973 , piece_id=X63L2GPVONW7ORT5DYOQIHV46EG4DIHEB6TLOZMXKLCQPR5EWDFA, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.817437797 (21h9m15.857010917s) [noise=true , address=45.91.116.11:4117 , piece_id=5E2JPHN5T2SETJNOGGAESLRU7PRPUCUDEO25QDW6PA73EKVGMRYQ, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.817462975 (21h9m15.857036096s) [seg_pos={0 0}, upload=18635]: Segment reached optimal threshold of 13 pieces. Sep 7 10:37:39.817597944 (21h9m15.857171064s) [noise=true , address=45.91.116.6:4975 , piece_id=7U2D53SCSJ5GUGLFWEH6FA66Q5U7EIK7YVBTP5GP5P5OOAVOHQWA, seg_pos={0 0}, upload=18635]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: expected piece hash; context canceled; EOF Sep 7 10:37:39.817683578 (21h9m15.857256698s) [noise=true , address=45.91.116.10:4083 , piece_id=2XGLF7IS7LC7TU5X2M27UEYQBVX3UML3UEWSFUISJOXF6AVKITTQ, seg_pos={0 0}, upload=18635]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: expected piece hash; context canceled; EOF Sep 7 10:37:39.817775434 (21h9m15.857348554s) [noise=true , address=45.91.116.12:4227 , piece_id=7BZ46KAIEIJDIKHXRLDCMB4Y6E7KSC4QLBOLY2A2NZFA4GVVXD2A, seg_pos={0 0}, upload=18635]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH): protocol: expected piece hash; context canceled; EOF Sep 7 10:37:39.817852270 (21h9m15.8574254s) [seg_pos={0 0}, upload=18635]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 7 10:37:39.817955087 (21h9m15.857528217s) [upload=18635]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 7 10:37:39.817961028 (21h9m15.857534148s) [upload=18635]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 7 10:37:40.298833319 (21h9m16.338406429s) [upload=18635]: Done uploading segments... Sep 7 10:37:40.298849160 (21h9m16.33842228s) [upload=18635]: Upload finished. err: <nil> Duration: 17.45224505s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.....X.................................................X....] duration: 5.995132615s FAIL (2025-09-07 10:41:18 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Sep 7 10:37:37.259810434 (21h9m13.299383554s) [upload=18635]: Starting upload Sep 7 10:37:37.259869848 (21h9m13.299442968s) [upload=18635]: Done starting upload Sep 7 10:37:37.259991862 (21h9m13.299564982s) [upload=18635]: Uploading segments... Sep 7 10:37:37.260091512 (21h9m13.299664632s) [upload=18635]: Got next segment. Inline: false Sep 7 10:37:37.260124284 (21h9m13.299697405s) [upload=18635]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 7 10:37:37.260129676 (21h9m13.299702796s) [upload=18635]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 7 10:37:37.523765418 (21h9m13.563338538s) [seg_pos={0 0}, upload=18635]: Begin upload segment... Sep 7 10:37:37.523848977 (21h9m13.563422098s) [seg_pos={0 0}, upload=18635]: Done begin upload segment. Sep 7 10:37:37.523875308 (21h9m13.563448418s) [noise=true , address=45.91.116.11:4119 , piece_id=Y5TR5QSARCJ3HXQZN6OWO7NTLMSGCSLCOEZVJRSB3VZQQL7O57NA, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.523927959 (21h9m13.563501079s) [noise=true , address=45.91.116.5:4298 , piece_id=T4KK43URSFHH2LDPBX4CN66G4KVBCWV7JSJM3CADPG6EYGOS2HFQ, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.524262138 (21h9m13.563835258s) [noise=true , address=45.91.116.6:4973 , piece_id=X63L2GPVONW7ORT5DYOQIHV46EG4DIHEB6TLOZMXKLCQPR5EWDFA, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.524469726 (21h9m13.564042846s) [noise=true , address=45.91.116.6:4974 , piece_id=ZUB4DD6ZXILUSYESNMGNOCKZVTPUY24N52YF2KY3Z36JYNDGRM2A, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.524553666 (21h9m13.564126786s) [noise=true , address=45.91.116.4:4939 , piece_id=BXLXHF3W5CPAS73RZXC4GL4PPN5PRMVLSSW4ADSW2M2KWRXQY77Q, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.524760001 (21h9m13.564333121s) [noise=true , address=77.90.31.10:4519 , piece_id=5RGPSBJO5NDUDD7Y4NH5CYTGF2FB5RO6ADKEP7YUGZFVMCTACVKA, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.524839723 (21h9m13.564412843s) [noise=true , address=45.91.116.12:4225 , piece_id=JQOKRSTWFE56HRHV4BBEL3ZR7T24XBHVYYU4WA4SWJGF432EPOLQ, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.525132163 (21h9m13.564705293s) [noise=true , address=45.91.116.12:4227 , piece_id=7BZ46KAIEIJDIKHXRLDCMB4Y6E7KSC4QLBOLY2A2NZFA4GVVXD2A, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.525296006 (21h9m13.564869127s) [noise=true , address=45.91.116.6:4975 , piece_id=7U2D53SCSJ5GUGLFWEH6FA66Q5U7EIK7YVBTP5GP5P5OOAVOHQWA, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.525386890 (21h9m13.56496001s) [noise=true , address=45.91.116.10:4083 , piece_id=2XGLF7IS7LC7TU5X2M27UEYQBVX3UML3UEWSFUISJOXF6AVKITTQ, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.525611531 (21h9m13.565184651s) [noise=true , address=45.91.116.7:4009 , piece_id=TYKXIJX7XIMFYKEA5RMORUFVO7UDVKNVBSJAIFUEIB3A76G4HN3Q, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.525648361 (21h9m13.565221481s) [noise=true , address=45.91.116.14:4191 , piece_id=BLZ4EZD5DQ7QQNNRLNCKAB2CQBZGQ2SYKGJJIOZCXNGS4SYOFNXA, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.525870506 (21h9m13.565443626s) [noise=true , address=45.91.116.11:4117 , piece_id=5E2JPHN5T2SETJNOGGAESLRU7PRPUCUDEO25QDW6PA73EKVGMRYQ, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.525935922 (21h9m13.565509042s) [noise=true , address=45.91.116.7:4010 , piece_id=UQRAOTS6AILKBAR6QC3ZD3VATJEYM2T6T4K2KDVU3XRKYQSEB66A, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.526222790 (21h9m13.56579591s) [noise=true , address=45.91.116.13:4153 , piece_id=KUSBCYGRA2XFXZPOYJG4EYGZ7FOSKPKV2EEFQHUND6TPTNQX7AGQ, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.526424746 (21h9m13.565997866s) [noise=true , address=77.90.31.10:4518 , piece_id=XBJTZKIXGIW3WPKU4RWYRZBMAC67YYDYRNAJNBWRYX6SHU3T73WA, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:39.362042598 (21h9m15.401615708s) [upload=18635]: Next returned nil segment Sep 7 10:37:39.362059750 (21h9m15.40163287s) [upload=18635]: Waiting for error group managing segments... Sep 7 10:37:39.669685089 (21h9m15.709258209s) [noise=true , address=45.91.116.14:4191 , piece_id=BLZ4EZD5DQ7QQNNRLNCKAB2CQBZGQ2SYKGJJIOZCXNGS4SYOFNXA, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.679437981 (21h9m15.719011101s) [noise=true , address=77.90.31.10:4519 , piece_id=5RGPSBJO5NDUDD7Y4NH5CYTGF2FB5RO6ADKEP7YUGZFVMCTACVKA, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.679555526 (21h9m15.719128646s) [noise=true , address=45.91.116.12:4225 , piece_id=JQOKRSTWFE56HRHV4BBEL3ZR7T24XBHVYYU4WA4SWJGF432EPOLQ, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.687220604 (21h9m15.726793724s) [noise=true , address=45.91.116.7:4010 , piece_id=UQRAOTS6AILKBAR6QC3ZD3VATJEYM2T6T4K2KDVU3XRKYQSEB66A, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.711092863 (21h9m15.750665983s) [noise=true , address=45.91.116.4:4939 , piece_id=BXLXHF3W5CPAS73RZXC4GL4PPN5PRMVLSSW4ADSW2M2KWRXQY77Q, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.734404882 (21h9m15.773978002s) [noise=true , address=45.91.116.7:4009 , piece_id=TYKXIJX7XIMFYKEA5RMORUFVO7UDVKNVBSJAIFUEIB3A76G4HN3Q, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.743143673 (21h9m15.782716793s) [noise=true , address=45.91.116.11:4119 , piece_id=Y5TR5QSARCJ3HXQZN6OWO7NTLMSGCSLCOEZVJRSB3VZQQL7O57NA, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.748574906 (21h9m15.788148016s) [noise=true , address=45.91.116.6:4974 , piece_id=ZUB4DD6ZXILUSYESNMGNOCKZVTPUY24N52YF2KY3Z36JYNDGRM2A, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.754311293 (21h9m15.793884413s) [noise=true , address=45.91.116.13:4153 , piece_id=KUSBCYGRA2XFXZPOYJG4EYGZ7FOSKPKV2EEFQHUND6TPTNQX7AGQ, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.755384276 (21h9m15.794957396s) [noise=true , address=45.91.116.5:4298 , piece_id=T4KK43URSFHH2LDPBX4CN66G4KVBCWV7JSJM3CADPG6EYGOS2HFQ, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.758010662 (21h9m15.797583782s) [noise=true , address=77.90.31.10:4518 , piece_id=XBJTZKIXGIW3WPKU4RWYRZBMAC67YYDYRNAJNBWRYX6SHU3T73WA, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.774473184 (21h9m15.814046304s) [noise=true , address=45.91.116.6:4973 , piece_id=X63L2GPVONW7ORT5DYOQIHV46EG4DIHEB6TLOZMXKLCQPR5EWDFA, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.817437797 (21h9m15.857010917s) [noise=true , address=45.91.116.11:4117 , piece_id=5E2JPHN5T2SETJNOGGAESLRU7PRPUCUDEO25QDW6PA73EKVGMRYQ, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.817462975 (21h9m15.857036096s) [seg_pos={0 0}, upload=18635]: Segment reached optimal threshold of 13 pieces. Sep 7 10:37:39.817597944 (21h9m15.857171064s) [noise=true , address=45.91.116.6:4975 , piece_id=7U2D53SCSJ5GUGLFWEH6FA66Q5U7EIK7YVBTP5GP5P5OOAVOHQWA, seg_pos={0 0}, upload=18635]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: expected piece hash; context canceled; EOF Sep 7 10:37:39.817683578 (21h9m15.857256698s) [noise=true , address=45.91.116.10:4083 , piece_id=2XGLF7IS7LC7TU5X2M27UEYQBVX3UML3UEWSFUISJOXF6AVKITTQ, seg_pos={0 0}, upload=18635]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: expected piece hash; context canceled; EOF Sep 7 10:37:39.817775434 (21h9m15.857348554s) [noise=true , address=45.91.116.12:4227 , piece_id=7BZ46KAIEIJDIKHXRLDCMB4Y6E7KSC4QLBOLY2A2NZFA4GVVXD2A, seg_pos={0 0}, upload=18635]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH): protocol: expected piece hash; context canceled; EOF Sep 7 10:37:39.817852270 (21h9m15.8574254s) [seg_pos={0 0}, upload=18635]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 7 10:37:39.817955087 (21h9m15.857528217s) [upload=18635]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 7 10:37:39.817961028 (21h9m15.857534148s) [upload=18635]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 7 10:37:40.298833319 (21h9m16.338406429s) [upload=18635]: Done uploading segments... Sep 7 10:37:40.298849160 (21h9m16.33842228s) [upload=18635]: Upload finished. err: <nil> Duration: 5.995161119s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [X.....X.................................................X...] duration: 7.091817428s FAIL (2025-09-07 10:40:20 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Sep 7 10:37:37.259810434 (21h9m13.299383554s) [upload=18635]: Starting upload Sep 7 10:37:37.259869848 (21h9m13.299442968s) [upload=18635]: Done starting upload Sep 7 10:37:37.259991862 (21h9m13.299564982s) [upload=18635]: Uploading segments... Sep 7 10:37:37.260091512 (21h9m13.299664632s) [upload=18635]: Got next segment. Inline: false Sep 7 10:37:37.260124284 (21h9m13.299697405s) [upload=18635]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 7 10:37:37.260129676 (21h9m13.299702796s) [upload=18635]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 7 10:37:37.523765418 (21h9m13.563338538s) [seg_pos={0 0}, upload=18635]: Begin upload segment... Sep 7 10:37:37.523848977 (21h9m13.563422098s) [seg_pos={0 0}, upload=18635]: Done begin upload segment. Sep 7 10:37:37.523875308 (21h9m13.563448418s) [noise=true , address=45.91.116.11:4119 , piece_id=Y5TR5QSARCJ3HXQZN6OWO7NTLMSGCSLCOEZVJRSB3VZQQL7O57NA, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.523927959 (21h9m13.563501079s) [noise=true , address=45.91.116.5:4298 , piece_id=T4KK43URSFHH2LDPBX4CN66G4KVBCWV7JSJM3CADPG6EYGOS2HFQ, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.524262138 (21h9m13.563835258s) [noise=true , address=45.91.116.6:4973 , piece_id=X63L2GPVONW7ORT5DYOQIHV46EG4DIHEB6TLOZMXKLCQPR5EWDFA, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.524469726 (21h9m13.564042846s) [noise=true , address=45.91.116.6:4974 , piece_id=ZUB4DD6ZXILUSYESNMGNOCKZVTPUY24N52YF2KY3Z36JYNDGRM2A, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.524553666 (21h9m13.564126786s) [noise=true , address=45.91.116.4:4939 , piece_id=BXLXHF3W5CPAS73RZXC4GL4PPN5PRMVLSSW4ADSW2M2KWRXQY77Q, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.524760001 (21h9m13.564333121s) [noise=true , address=77.90.31.10:4519 , piece_id=5RGPSBJO5NDUDD7Y4NH5CYTGF2FB5RO6ADKEP7YUGZFVMCTACVKA, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.524839723 (21h9m13.564412843s) [noise=true , address=45.91.116.12:4225 , piece_id=JQOKRSTWFE56HRHV4BBEL3ZR7T24XBHVYYU4WA4SWJGF432EPOLQ, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.525132163 (21h9m13.564705293s) [noise=true , address=45.91.116.12:4227 , piece_id=7BZ46KAIEIJDIKHXRLDCMB4Y6E7KSC4QLBOLY2A2NZFA4GVVXD2A, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.525296006 (21h9m13.564869127s) [noise=true , address=45.91.116.6:4975 , piece_id=7U2D53SCSJ5GUGLFWEH6FA66Q5U7EIK7YVBTP5GP5P5OOAVOHQWA, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.525386890 (21h9m13.56496001s) [noise=true , address=45.91.116.10:4083 , piece_id=2XGLF7IS7LC7TU5X2M27UEYQBVX3UML3UEWSFUISJOXF6AVKITTQ, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.525611531 (21h9m13.565184651s) [noise=true , address=45.91.116.7:4009 , piece_id=TYKXIJX7XIMFYKEA5RMORUFVO7UDVKNVBSJAIFUEIB3A76G4HN3Q, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.525648361 (21h9m13.565221481s) [noise=true , address=45.91.116.14:4191 , piece_id=BLZ4EZD5DQ7QQNNRLNCKAB2CQBZGQ2SYKGJJIOZCXNGS4SYOFNXA, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.525870506 (21h9m13.565443626s) [noise=true , address=45.91.116.11:4117 , piece_id=5E2JPHN5T2SETJNOGGAESLRU7PRPUCUDEO25QDW6PA73EKVGMRYQ, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.525935922 (21h9m13.565509042s) [noise=true , address=45.91.116.7:4010 , piece_id=UQRAOTS6AILKBAR6QC3ZD3VATJEYM2T6T4K2KDVU3XRKYQSEB66A, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.526222790 (21h9m13.56579591s) [noise=true , address=45.91.116.13:4153 , piece_id=KUSBCYGRA2XFXZPOYJG4EYGZ7FOSKPKV2EEFQHUND6TPTNQX7AGQ, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.526424746 (21h9m13.565997866s) [noise=true , address=77.90.31.10:4518 , piece_id=XBJTZKIXGIW3WPKU4RWYRZBMAC67YYDYRNAJNBWRYX6SHU3T73WA, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:39.362042598 (21h9m15.401615708s) [upload=18635]: Next returned nil segment Sep 7 10:37:39.362059750 (21h9m15.40163287s) [upload=18635]: Waiting for error group managing segments... Sep 7 10:37:39.669685089 (21h9m15.709258209s) [noise=true , address=45.91.116.14:4191 , piece_id=BLZ4EZD5DQ7QQNNRLNCKAB2CQBZGQ2SYKGJJIOZCXNGS4SYOFNXA, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.679437981 (21h9m15.719011101s) [noise=true , address=77.90.31.10:4519 , piece_id=5RGPSBJO5NDUDD7Y4NH5CYTGF2FB5RO6ADKEP7YUGZFVMCTACVKA, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.679555526 (21h9m15.719128646s) [noise=true , address=45.91.116.12:4225 , piece_id=JQOKRSTWFE56HRHV4BBEL3ZR7T24XBHVYYU4WA4SWJGF432EPOLQ, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.687220604 (21h9m15.726793724s) [noise=true , address=45.91.116.7:4010 , piece_id=UQRAOTS6AILKBAR6QC3ZD3VATJEYM2T6T4K2KDVU3XRKYQSEB66A, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.711092863 (21h9m15.750665983s) [noise=true , address=45.91.116.4:4939 , piece_id=BXLXHF3W5CPAS73RZXC4GL4PPN5PRMVLSSW4ADSW2M2KWRXQY77Q, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.734404882 (21h9m15.773978002s) [noise=true , address=45.91.116.7:4009 , piece_id=TYKXIJX7XIMFYKEA5RMORUFVO7UDVKNVBSJAIFUEIB3A76G4HN3Q, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.743143673 (21h9m15.782716793s) [noise=true , address=45.91.116.11:4119 , piece_id=Y5TR5QSARCJ3HXQZN6OWO7NTLMSGCSLCOEZVJRSB3VZQQL7O57NA, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.748574906 (21h9m15.788148016s) [noise=true , address=45.91.116.6:4974 , piece_id=ZUB4DD6ZXILUSYESNMGNOCKZVTPUY24N52YF2KY3Z36JYNDGRM2A, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.754311293 (21h9m15.793884413s) [noise=true , address=45.91.116.13:4153 , piece_id=KUSBCYGRA2XFXZPOYJG4EYGZ7FOSKPKV2EEFQHUND6TPTNQX7AGQ, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.755384276 (21h9m15.794957396s) [noise=true , address=45.91.116.5:4298 , piece_id=T4KK43URSFHH2LDPBX4CN66G4KVBCWV7JSJM3CADPG6EYGOS2HFQ, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.758010662 (21h9m15.797583782s) [noise=true , address=77.90.31.10:4518 , piece_id=XBJTZKIXGIW3WPKU4RWYRZBMAC67YYDYRNAJNBWRYX6SHU3T73WA, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.774473184 (21h9m15.814046304s) [noise=true , address=45.91.116.6:4973 , piece_id=X63L2GPVONW7ORT5DYOQIHV46EG4DIHEB6TLOZMXKLCQPR5EWDFA, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.817437797 (21h9m15.857010917s) [noise=true , address=45.91.116.11:4117 , piece_id=5E2JPHN5T2SETJNOGGAESLRU7PRPUCUDEO25QDW6PA73EKVGMRYQ, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.817462975 (21h9m15.857036096s) [seg_pos={0 0}, upload=18635]: Segment reached optimal threshold of 13 pieces. Sep 7 10:37:39.817597944 (21h9m15.857171064s) [noise=true , address=45.91.116.6:4975 , piece_id=7U2D53SCSJ5GUGLFWEH6FA66Q5U7EIK7YVBTP5GP5P5OOAVOHQWA, seg_pos={0 0}, upload=18635]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: expected piece hash; context canceled; EOF Sep 7 10:37:39.817683578 (21h9m15.857256698s) [noise=true , address=45.91.116.10:4083 , piece_id=2XGLF7IS7LC7TU5X2M27UEYQBVX3UML3UEWSFUISJOXF6AVKITTQ, seg_pos={0 0}, upload=18635]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: expected piece hash; context canceled; EOF Sep 7 10:37:39.817775434 (21h9m15.857348554s) [noise=true , address=45.91.116.12:4227 , piece_id=7BZ46KAIEIJDIKHXRLDCMB4Y6E7KSC4QLBOLY2A2NZFA4GVVXD2A, seg_pos={0 0}, upload=18635]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH): protocol: expected piece hash; context canceled; EOF Sep 7 10:37:39.817852270 (21h9m15.8574254s) [seg_pos={0 0}, upload=18635]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 7 10:37:39.817955087 (21h9m15.857528217s) [upload=18635]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 7 10:37:39.817961028 (21h9m15.857534148s) [upload=18635]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 7 10:37:40.298833319 (21h9m16.338406429s) [upload=18635]: Done uploading segments... Sep 7 10:37:40.298849160 (21h9m16.33842228s) [upload=18635]: Upload finished. err: <nil> Duration: 7.09184005s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.X.....X.................................................X..] duration: 6.957865609s FAIL (2025-09-07 10:39:08 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Sep 7 10:37:37.259810434 (21h9m13.299383554s) [upload=18635]: Starting upload Sep 7 10:37:37.259869848 (21h9m13.299442968s) [upload=18635]: Done starting upload Sep 7 10:37:37.259991862 (21h9m13.299564982s) [upload=18635]: Uploading segments... Sep 7 10:37:37.260091512 (21h9m13.299664632s) [upload=18635]: Got next segment. Inline: false Sep 7 10:37:37.260124284 (21h9m13.299697405s) [upload=18635]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 7 10:37:37.260129676 (21h9m13.299702796s) [upload=18635]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 7 10:37:37.523765418 (21h9m13.563338538s) [seg_pos={0 0}, upload=18635]: Begin upload segment... Sep 7 10:37:37.523848977 (21h9m13.563422098s) [seg_pos={0 0}, upload=18635]: Done begin upload segment. Sep 7 10:37:37.523875308 (21h9m13.563448418s) [noise=true , address=45.91.116.11:4119 , piece_id=Y5TR5QSARCJ3HXQZN6OWO7NTLMSGCSLCOEZVJRSB3VZQQL7O57NA, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.523927959 (21h9m13.563501079s) [noise=true , address=45.91.116.5:4298 , piece_id=T4KK43URSFHH2LDPBX4CN66G4KVBCWV7JSJM3CADPG6EYGOS2HFQ, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.524262138 (21h9m13.563835258s) [noise=true , address=45.91.116.6:4973 , piece_id=X63L2GPVONW7ORT5DYOQIHV46EG4DIHEB6TLOZMXKLCQPR5EWDFA, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.524469726 (21h9m13.564042846s) [noise=true , address=45.91.116.6:4974 , piece_id=ZUB4DD6ZXILUSYESNMGNOCKZVTPUY24N52YF2KY3Z36JYNDGRM2A, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.524553666 (21h9m13.564126786s) [noise=true , address=45.91.116.4:4939 , piece_id=BXLXHF3W5CPAS73RZXC4GL4PPN5PRMVLSSW4ADSW2M2KWRXQY77Q, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.524760001 (21h9m13.564333121s) [noise=true , address=77.90.31.10:4519 , piece_id=5RGPSBJO5NDUDD7Y4NH5CYTGF2FB5RO6ADKEP7YUGZFVMCTACVKA, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.524839723 (21h9m13.564412843s) [noise=true , address=45.91.116.12:4225 , piece_id=JQOKRSTWFE56HRHV4BBEL3ZR7T24XBHVYYU4WA4SWJGF432EPOLQ, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.525132163 (21h9m13.564705293s) [noise=true , address=45.91.116.12:4227 , piece_id=7BZ46KAIEIJDIKHXRLDCMB4Y6E7KSC4QLBOLY2A2NZFA4GVVXD2A, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.525296006 (21h9m13.564869127s) [noise=true , address=45.91.116.6:4975 , piece_id=7U2D53SCSJ5GUGLFWEH6FA66Q5U7EIK7YVBTP5GP5P5OOAVOHQWA, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.525386890 (21h9m13.56496001s) [noise=true , address=45.91.116.10:4083 , piece_id=2XGLF7IS7LC7TU5X2M27UEYQBVX3UML3UEWSFUISJOXF6AVKITTQ, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.525611531 (21h9m13.565184651s) [noise=true , address=45.91.116.7:4009 , piece_id=TYKXIJX7XIMFYKEA5RMORUFVO7UDVKNVBSJAIFUEIB3A76G4HN3Q, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.525648361 (21h9m13.565221481s) [noise=true , address=45.91.116.14:4191 , piece_id=BLZ4EZD5DQ7QQNNRLNCKAB2CQBZGQ2SYKGJJIOZCXNGS4SYOFNXA, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.525870506 (21h9m13.565443626s) [noise=true , address=45.91.116.11:4117 , piece_id=5E2JPHN5T2SETJNOGGAESLRU7PRPUCUDEO25QDW6PA73EKVGMRYQ, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.525935922 (21h9m13.565509042s) [noise=true , address=45.91.116.7:4010 , piece_id=UQRAOTS6AILKBAR6QC3ZD3VATJEYM2T6T4K2KDVU3XRKYQSEB66A, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.526222790 (21h9m13.56579591s) [noise=true , address=45.91.116.13:4153 , piece_id=KUSBCYGRA2XFXZPOYJG4EYGZ7FOSKPKV2EEFQHUND6TPTNQX7AGQ, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.526424746 (21h9m13.565997866s) [noise=true , address=77.90.31.10:4518 , piece_id=XBJTZKIXGIW3WPKU4RWYRZBMAC67YYDYRNAJNBWRYX6SHU3T73WA, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:39.362042598 (21h9m15.401615708s) [upload=18635]: Next returned nil segment Sep 7 10:37:39.362059750 (21h9m15.40163287s) [upload=18635]: Waiting for error group managing segments... Sep 7 10:37:39.669685089 (21h9m15.709258209s) [noise=true , address=45.91.116.14:4191 , piece_id=BLZ4EZD5DQ7QQNNRLNCKAB2CQBZGQ2SYKGJJIOZCXNGS4SYOFNXA, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.679437981 (21h9m15.719011101s) [noise=true , address=77.90.31.10:4519 , piece_id=5RGPSBJO5NDUDD7Y4NH5CYTGF2FB5RO6ADKEP7YUGZFVMCTACVKA, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.679555526 (21h9m15.719128646s) [noise=true , address=45.91.116.12:4225 , piece_id=JQOKRSTWFE56HRHV4BBEL3ZR7T24XBHVYYU4WA4SWJGF432EPOLQ, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.687220604 (21h9m15.726793724s) [noise=true , address=45.91.116.7:4010 , piece_id=UQRAOTS6AILKBAR6QC3ZD3VATJEYM2T6T4K2KDVU3XRKYQSEB66A, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.711092863 (21h9m15.750665983s) [noise=true , address=45.91.116.4:4939 , piece_id=BXLXHF3W5CPAS73RZXC4GL4PPN5PRMVLSSW4ADSW2M2KWRXQY77Q, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.734404882 (21h9m15.773978002s) [noise=true , address=45.91.116.7:4009 , piece_id=TYKXIJX7XIMFYKEA5RMORUFVO7UDVKNVBSJAIFUEIB3A76G4HN3Q, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.743143673 (21h9m15.782716793s) [noise=true , address=45.91.116.11:4119 , piece_id=Y5TR5QSARCJ3HXQZN6OWO7NTLMSGCSLCOEZVJRSB3VZQQL7O57NA, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.748574906 (21h9m15.788148016s) [noise=true , address=45.91.116.6:4974 , piece_id=ZUB4DD6ZXILUSYESNMGNOCKZVTPUY24N52YF2KY3Z36JYNDGRM2A, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.754311293 (21h9m15.793884413s) [noise=true , address=45.91.116.13:4153 , piece_id=KUSBCYGRA2XFXZPOYJG4EYGZ7FOSKPKV2EEFQHUND6TPTNQX7AGQ, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.755384276 (21h9m15.794957396s) [noise=true , address=45.91.116.5:4298 , piece_id=T4KK43URSFHH2LDPBX4CN66G4KVBCWV7JSJM3CADPG6EYGOS2HFQ, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.758010662 (21h9m15.797583782s) [noise=true , address=77.90.31.10:4518 , piece_id=XBJTZKIXGIW3WPKU4RWYRZBMAC67YYDYRNAJNBWRYX6SHU3T73WA, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.774473184 (21h9m15.814046304s) [noise=true , address=45.91.116.6:4973 , piece_id=X63L2GPVONW7ORT5DYOQIHV46EG4DIHEB6TLOZMXKLCQPR5EWDFA, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.817437797 (21h9m15.857010917s) [noise=true , address=45.91.116.11:4117 , piece_id=5E2JPHN5T2SETJNOGGAESLRU7PRPUCUDEO25QDW6PA73EKVGMRYQ, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.817462975 (21h9m15.857036096s) [seg_pos={0 0}, upload=18635]: Segment reached optimal threshold of 13 pieces. Sep 7 10:37:39.817597944 (21h9m15.857171064s) [noise=true , address=45.91.116.6:4975 , piece_id=7U2D53SCSJ5GUGLFWEH6FA66Q5U7EIK7YVBTP5GP5P5OOAVOHQWA, seg_pos={0 0}, upload=18635]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: expected piece hash; context canceled; EOF Sep 7 10:37:39.817683578 (21h9m15.857256698s) [noise=true , address=45.91.116.10:4083 , piece_id=2XGLF7IS7LC7TU5X2M27UEYQBVX3UML3UEWSFUISJOXF6AVKITTQ, seg_pos={0 0}, upload=18635]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: expected piece hash; context canceled; EOF Sep 7 10:37:39.817775434 (21h9m15.857348554s) [noise=true , address=45.91.116.12:4227 , piece_id=7BZ46KAIEIJDIKHXRLDCMB4Y6E7KSC4QLBOLY2A2NZFA4GVVXD2A, seg_pos={0 0}, upload=18635]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH): protocol: expected piece hash; context canceled; EOF Sep 7 10:37:39.817852270 (21h9m15.8574254s) [seg_pos={0 0}, upload=18635]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 7 10:37:39.817955087 (21h9m15.857528217s) [upload=18635]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 7 10:37:39.817961028 (21h9m15.857534148s) [upload=18635]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 7 10:37:40.298833319 (21h9m16.338406429s) [upload=18635]: Done uploading segments... Sep 7 10:37:40.298849160 (21h9m16.33842228s) [upload=18635]: Upload finished. err: <nil> Duration: 6.957894404s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [..X.....X.................................................X.] duration: 11.928795904s FAIL (2025-09-07 10:38:28 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Sep 7 10:37:37.259810434 (21h9m13.299383554s) [upload=18635]: Starting upload Sep 7 10:37:37.259869848 (21h9m13.299442968s) [upload=18635]: Done starting upload Sep 7 10:37:37.259991862 (21h9m13.299564982s) [upload=18635]: Uploading segments... Sep 7 10:37:37.260091512 (21h9m13.299664632s) [upload=18635]: Got next segment. Inline: false Sep 7 10:37:37.260124284 (21h9m13.299697405s) [upload=18635]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 7 10:37:37.260129676 (21h9m13.299702796s) [upload=18635]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 7 10:37:37.523765418 (21h9m13.563338538s) [seg_pos={0 0}, upload=18635]: Begin upload segment... Sep 7 10:37:37.523848977 (21h9m13.563422098s) [seg_pos={0 0}, upload=18635]: Done begin upload segment. Sep 7 10:37:37.523875308 (21h9m13.563448418s) [noise=true , address=45.91.116.11:4119 , piece_id=Y5TR5QSARCJ3HXQZN6OWO7NTLMSGCSLCOEZVJRSB3VZQQL7O57NA, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.523927959 (21h9m13.563501079s) [noise=true , address=45.91.116.5:4298 , piece_id=T4KK43URSFHH2LDPBX4CN66G4KVBCWV7JSJM3CADPG6EYGOS2HFQ, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.524262138 (21h9m13.563835258s) [noise=true , address=45.91.116.6:4973 , piece_id=X63L2GPVONW7ORT5DYOQIHV46EG4DIHEB6TLOZMXKLCQPR5EWDFA, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.524469726 (21h9m13.564042846s) [noise=true , address=45.91.116.6:4974 , piece_id=ZUB4DD6ZXILUSYESNMGNOCKZVTPUY24N52YF2KY3Z36JYNDGRM2A, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.524553666 (21h9m13.564126786s) [noise=true , address=45.91.116.4:4939 , piece_id=BXLXHF3W5CPAS73RZXC4GL4PPN5PRMVLSSW4ADSW2M2KWRXQY77Q, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.524760001 (21h9m13.564333121s) [noise=true , address=77.90.31.10:4519 , piece_id=5RGPSBJO5NDUDD7Y4NH5CYTGF2FB5RO6ADKEP7YUGZFVMCTACVKA, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.524839723 (21h9m13.564412843s) [noise=true , address=45.91.116.12:4225 , piece_id=JQOKRSTWFE56HRHV4BBEL3ZR7T24XBHVYYU4WA4SWJGF432EPOLQ, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.525132163 (21h9m13.564705293s) [noise=true , address=45.91.116.12:4227 , piece_id=7BZ46KAIEIJDIKHXRLDCMB4Y6E7KSC4QLBOLY2A2NZFA4GVVXD2A, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.525296006 (21h9m13.564869127s) [noise=true , address=45.91.116.6:4975 , piece_id=7U2D53SCSJ5GUGLFWEH6FA66Q5U7EIK7YVBTP5GP5P5OOAVOHQWA, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.525386890 (21h9m13.56496001s) [noise=true , address=45.91.116.10:4083 , piece_id=2XGLF7IS7LC7TU5X2M27UEYQBVX3UML3UEWSFUISJOXF6AVKITTQ, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.525611531 (21h9m13.565184651s) [noise=true , address=45.91.116.7:4009 , piece_id=TYKXIJX7XIMFYKEA5RMORUFVO7UDVKNVBSJAIFUEIB3A76G4HN3Q, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.525648361 (21h9m13.565221481s) [noise=true , address=45.91.116.14:4191 , piece_id=BLZ4EZD5DQ7QQNNRLNCKAB2CQBZGQ2SYKGJJIOZCXNGS4SYOFNXA, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.525870506 (21h9m13.565443626s) [noise=true , address=45.91.116.11:4117 , piece_id=5E2JPHN5T2SETJNOGGAESLRU7PRPUCUDEO25QDW6PA73EKVGMRYQ, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.525935922 (21h9m13.565509042s) [noise=true , address=45.91.116.7:4010 , piece_id=UQRAOTS6AILKBAR6QC3ZD3VATJEYM2T6T4K2KDVU3XRKYQSEB66A, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.526222790 (21h9m13.56579591s) [noise=true , address=45.91.116.13:4153 , piece_id=KUSBCYGRA2XFXZPOYJG4EYGZ7FOSKPKV2EEFQHUND6TPTNQX7AGQ, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.526424746 (21h9m13.565997866s) [noise=true , address=77.90.31.10:4518 , piece_id=XBJTZKIXGIW3WPKU4RWYRZBMAC67YYDYRNAJNBWRYX6SHU3T73WA, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:39.362042598 (21h9m15.401615708s) [upload=18635]: Next returned nil segment Sep 7 10:37:39.362059750 (21h9m15.40163287s) [upload=18635]: Waiting for error group managing segments... Sep 7 10:37:39.669685089 (21h9m15.709258209s) [noise=true , address=45.91.116.14:4191 , piece_id=BLZ4EZD5DQ7QQNNRLNCKAB2CQBZGQ2SYKGJJIOZCXNGS4SYOFNXA, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.679437981 (21h9m15.719011101s) [noise=true , address=77.90.31.10:4519 , piece_id=5RGPSBJO5NDUDD7Y4NH5CYTGF2FB5RO6ADKEP7YUGZFVMCTACVKA, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.679555526 (21h9m15.719128646s) [noise=true , address=45.91.116.12:4225 , piece_id=JQOKRSTWFE56HRHV4BBEL3ZR7T24XBHVYYU4WA4SWJGF432EPOLQ, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.687220604 (21h9m15.726793724s) [noise=true , address=45.91.116.7:4010 , piece_id=UQRAOTS6AILKBAR6QC3ZD3VATJEYM2T6T4K2KDVU3XRKYQSEB66A, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.711092863 (21h9m15.750665983s) [noise=true , address=45.91.116.4:4939 , piece_id=BXLXHF3W5CPAS73RZXC4GL4PPN5PRMVLSSW4ADSW2M2KWRXQY77Q, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.734404882 (21h9m15.773978002s) [noise=true , address=45.91.116.7:4009 , piece_id=TYKXIJX7XIMFYKEA5RMORUFVO7UDVKNVBSJAIFUEIB3A76G4HN3Q, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.743143673 (21h9m15.782716793s) [noise=true , address=45.91.116.11:4119 , piece_id=Y5TR5QSARCJ3HXQZN6OWO7NTLMSGCSLCOEZVJRSB3VZQQL7O57NA, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.748574906 (21h9m15.788148016s) [noise=true , address=45.91.116.6:4974 , piece_id=ZUB4DD6ZXILUSYESNMGNOCKZVTPUY24N52YF2KY3Z36JYNDGRM2A, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.754311293 (21h9m15.793884413s) [noise=true , address=45.91.116.13:4153 , piece_id=KUSBCYGRA2XFXZPOYJG4EYGZ7FOSKPKV2EEFQHUND6TPTNQX7AGQ, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.755384276 (21h9m15.794957396s) [noise=true , address=45.91.116.5:4298 , piece_id=T4KK43URSFHH2LDPBX4CN66G4KVBCWV7JSJM3CADPG6EYGOS2HFQ, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.758010662 (21h9m15.797583782s) [noise=true , address=77.90.31.10:4518 , piece_id=XBJTZKIXGIW3WPKU4RWYRZBMAC67YYDYRNAJNBWRYX6SHU3T73WA, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.774473184 (21h9m15.814046304s) [noise=true , address=45.91.116.6:4973 , piece_id=X63L2GPVONW7ORT5DYOQIHV46EG4DIHEB6TLOZMXKLCQPR5EWDFA, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.817437797 (21h9m15.857010917s) [noise=true , address=45.91.116.11:4117 , piece_id=5E2JPHN5T2SETJNOGGAESLRU7PRPUCUDEO25QDW6PA73EKVGMRYQ, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.817462975 (21h9m15.857036096s) [seg_pos={0 0}, upload=18635]: Segment reached optimal threshold of 13 pieces. Sep 7 10:37:39.817597944 (21h9m15.857171064s) [noise=true , address=45.91.116.6:4975 , piece_id=7U2D53SCSJ5GUGLFWEH6FA66Q5U7EIK7YVBTP5GP5P5OOAVOHQWA, seg_pos={0 0}, upload=18635]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: expected piece hash; context canceled; EOF Sep 7 10:37:39.817683578 (21h9m15.857256698s) [noise=true , address=45.91.116.10:4083 , piece_id=2XGLF7IS7LC7TU5X2M27UEYQBVX3UML3UEWSFUISJOXF6AVKITTQ, seg_pos={0 0}, upload=18635]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: expected piece hash; context canceled; EOF Sep 7 10:37:39.817775434 (21h9m15.857348554s) [noise=true , address=45.91.116.12:4227 , piece_id=7BZ46KAIEIJDIKHXRLDCMB4Y6E7KSC4QLBOLY2A2NZFA4GVVXD2A, seg_pos={0 0}, upload=18635]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH): protocol: expected piece hash; context canceled; EOF Sep 7 10:37:39.817852270 (21h9m15.8574254s) [seg_pos={0 0}, upload=18635]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 7 10:37:39.817955087 (21h9m15.857528217s) [upload=18635]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 7 10:37:39.817961028 (21h9m15.857534148s) [upload=18635]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 7 10:37:40.298833319 (21h9m16.338406429s) [upload=18635]: Done uploading segments... Sep 7 10:37:40.298849160 (21h9m16.33842228s) [upload=18635]: Upload finished. err: <nil> Duration: 11.928821404s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [...X.....X.................................................X] duration: 30.001358879s FAIL (2025-09-07 10:38:06 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Sep 7 10:37:37.259810434 (21h9m13.299383554s) [upload=18635]: Starting upload Sep 7 10:37:37.259869848 (21h9m13.299442968s) [upload=18635]: Done starting upload Sep 7 10:37:37.259991862 (21h9m13.299564982s) [upload=18635]: Uploading segments... Sep 7 10:37:37.260091512 (21h9m13.299664632s) [upload=18635]: Got next segment. Inline: false Sep 7 10:37:37.260124284 (21h9m13.299697405s) [upload=18635]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 7 10:37:37.260129676 (21h9m13.299702796s) [upload=18635]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 7 10:37:37.523765418 (21h9m13.563338538s) [seg_pos={0 0}, upload=18635]: Begin upload segment... Sep 7 10:37:37.523848977 (21h9m13.563422098s) [seg_pos={0 0}, upload=18635]: Done begin upload segment. Sep 7 10:37:37.523875308 (21h9m13.563448418s) [noise=true , address=45.91.116.11:4119 , piece_id=Y5TR5QSARCJ3HXQZN6OWO7NTLMSGCSLCOEZVJRSB3VZQQL7O57NA, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.523927959 (21h9m13.563501079s) [noise=true , address=45.91.116.5:4298 , piece_id=T4KK43URSFHH2LDPBX4CN66G4KVBCWV7JSJM3CADPG6EYGOS2HFQ, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.524262138 (21h9m13.563835258s) [noise=true , address=45.91.116.6:4973 , piece_id=X63L2GPVONW7ORT5DYOQIHV46EG4DIHEB6TLOZMXKLCQPR5EWDFA, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.524469726 (21h9m13.564042846s) [noise=true , address=45.91.116.6:4974 , piece_id=ZUB4DD6ZXILUSYESNMGNOCKZVTPUY24N52YF2KY3Z36JYNDGRM2A, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.524553666 (21h9m13.564126786s) [noise=true , address=45.91.116.4:4939 , piece_id=BXLXHF3W5CPAS73RZXC4GL4PPN5PRMVLSSW4ADSW2M2KWRXQY77Q, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.524760001 (21h9m13.564333121s) [noise=true , address=77.90.31.10:4519 , piece_id=5RGPSBJO5NDUDD7Y4NH5CYTGF2FB5RO6ADKEP7YUGZFVMCTACVKA, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.524839723 (21h9m13.564412843s) [noise=true , address=45.91.116.12:4225 , piece_id=JQOKRSTWFE56HRHV4BBEL3ZR7T24XBHVYYU4WA4SWJGF432EPOLQ, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.525132163 (21h9m13.564705293s) [noise=true , address=45.91.116.12:4227 , piece_id=7BZ46KAIEIJDIKHXRLDCMB4Y6E7KSC4QLBOLY2A2NZFA4GVVXD2A, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.525296006 (21h9m13.564869127s) [noise=true , address=45.91.116.6:4975 , piece_id=7U2D53SCSJ5GUGLFWEH6FA66Q5U7EIK7YVBTP5GP5P5OOAVOHQWA, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.525386890 (21h9m13.56496001s) [noise=true , address=45.91.116.10:4083 , piece_id=2XGLF7IS7LC7TU5X2M27UEYQBVX3UML3UEWSFUISJOXF6AVKITTQ, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.525611531 (21h9m13.565184651s) [noise=true , address=45.91.116.7:4009 , piece_id=TYKXIJX7XIMFYKEA5RMORUFVO7UDVKNVBSJAIFUEIB3A76G4HN3Q, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.525648361 (21h9m13.565221481s) [noise=true , address=45.91.116.14:4191 , piece_id=BLZ4EZD5DQ7QQNNRLNCKAB2CQBZGQ2SYKGJJIOZCXNGS4SYOFNXA, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.525870506 (21h9m13.565443626s) [noise=true , address=45.91.116.11:4117 , piece_id=5E2JPHN5T2SETJNOGGAESLRU7PRPUCUDEO25QDW6PA73EKVGMRYQ, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.525935922 (21h9m13.565509042s) [noise=true , address=45.91.116.7:4010 , piece_id=UQRAOTS6AILKBAR6QC3ZD3VATJEYM2T6T4K2KDVU3XRKYQSEB66A, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.526222790 (21h9m13.56579591s) [noise=true , address=45.91.116.13:4153 , piece_id=KUSBCYGRA2XFXZPOYJG4EYGZ7FOSKPKV2EEFQHUND6TPTNQX7AGQ, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:37.526424746 (21h9m13.565997866s) [noise=true , address=77.90.31.10:4518 , piece_id=XBJTZKIXGIW3WPKU4RWYRZBMAC67YYDYRNAJNBWRYX6SHU3T73WA, seg_pos={0 0}, upload=18635]: Uploading piece... Sep 7 10:37:39.362042598 (21h9m15.401615708s) [upload=18635]: Next returned nil segment Sep 7 10:37:39.362059750 (21h9m15.40163287s) [upload=18635]: Waiting for error group managing segments... Sep 7 10:37:39.669685089 (21h9m15.709258209s) [noise=true , address=45.91.116.14:4191 , piece_id=BLZ4EZD5DQ7QQNNRLNCKAB2CQBZGQ2SYKGJJIOZCXNGS4SYOFNXA, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.679437981 (21h9m15.719011101s) [noise=true , address=77.90.31.10:4519 , piece_id=5RGPSBJO5NDUDD7Y4NH5CYTGF2FB5RO6ADKEP7YUGZFVMCTACVKA, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.679555526 (21h9m15.719128646s) [noise=true , address=45.91.116.12:4225 , piece_id=JQOKRSTWFE56HRHV4BBEL3ZR7T24XBHVYYU4WA4SWJGF432EPOLQ, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.687220604 (21h9m15.726793724s) [noise=true , address=45.91.116.7:4010 , piece_id=UQRAOTS6AILKBAR6QC3ZD3VATJEYM2T6T4K2KDVU3XRKYQSEB66A, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.711092863 (21h9m15.750665983s) [noise=true , address=45.91.116.4:4939 , piece_id=BXLXHF3W5CPAS73RZXC4GL4PPN5PRMVLSSW4ADSW2M2KWRXQY77Q, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.734404882 (21h9m15.773978002s) [noise=true , address=45.91.116.7:4009 , piece_id=TYKXIJX7XIMFYKEA5RMORUFVO7UDVKNVBSJAIFUEIB3A76G4HN3Q, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.743143673 (21h9m15.782716793s) [noise=true , address=45.91.116.11:4119 , piece_id=Y5TR5QSARCJ3HXQZN6OWO7NTLMSGCSLCOEZVJRSB3VZQQL7O57NA, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.748574906 (21h9m15.788148016s) [noise=true , address=45.91.116.6:4974 , piece_id=ZUB4DD6ZXILUSYESNMGNOCKZVTPUY24N52YF2KY3Z36JYNDGRM2A, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.754311293 (21h9m15.793884413s) [noise=true , address=45.91.116.13:4153 , piece_id=KUSBCYGRA2XFXZPOYJG4EYGZ7FOSKPKV2EEFQHUND6TPTNQX7AGQ, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.755384276 (21h9m15.794957396s) [noise=true , address=45.91.116.5:4298 , piece_id=T4KK43URSFHH2LDPBX4CN66G4KVBCWV7JSJM3CADPG6EYGOS2HFQ, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.758010662 (21h9m15.797583782s) [noise=true , address=77.90.31.10:4518 , piece_id=XBJTZKIXGIW3WPKU4RWYRZBMAC67YYDYRNAJNBWRYX6SHU3T73WA, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.774473184 (21h9m15.814046304s) [noise=true , address=45.91.116.6:4973 , piece_id=X63L2GPVONW7ORT5DYOQIHV46EG4DIHEB6TLOZMXKLCQPR5EWDFA, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.817437797 (21h9m15.857010917s) [noise=true , address=45.91.116.11:4117 , piece_id=5E2JPHN5T2SETJNOGGAESLRU7PRPUCUDEO25QDW6PA73EKVGMRYQ, seg_pos={0 0}, upload=18635]: Done uploading piece. err: <nil> Sep 7 10:37:39.817462975 (21h9m15.857036096s) [seg_pos={0 0}, upload=18635]: Segment reached optimal threshold of 13 pieces. Sep 7 10:37:39.817597944 (21h9m15.857171064s) [noise=true , address=45.91.116.6:4975 , piece_id=7U2D53SCSJ5GUGLFWEH6FA66Q5U7EIK7YVBTP5GP5P5OOAVOHQWA, seg_pos={0 0}, upload=18635]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: expected piece hash; context canceled; EOF Sep 7 10:37:39.817683578 (21h9m15.857256698s) [noise=true , address=45.91.116.10:4083 , piece_id=2XGLF7IS7LC7TU5X2M27UEYQBVX3UML3UEWSFUISJOXF6AVKITTQ, seg_pos={0 0}, upload=18635]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: expected piece hash; context canceled; EOF Sep 7 10:37:39.817775434 (21h9m15.857348554s) [noise=true , address=45.91.116.12:4227 , piece_id=7BZ46KAIEIJDIKHXRLDCMB4Y6E7KSC4QLBOLY2A2NZFA4GVVXD2A, seg_pos={0 0}, upload=18635]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH): protocol: expected piece hash; context canceled; EOF Sep 7 10:37:39.817852270 (21h9m15.8574254s) [seg_pos={0 0}, upload=18635]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 7 10:37:39.817955087 (21h9m15.857528217s) [upload=18635]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 7 10:37:39.817961028 (21h9m15.857534148s) [upload=18635]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 7 10:37:40.298833319 (21h9m16.338406429s) [upload=18635]: Done uploading segments... Sep 7 10:37:40.298849160 (21h9m16.33842228s) [upload=18635]: Upload finished. err: <nil> Duration: 30.001380692s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [....X.....X.................................................] duration: 7.260928425s FAIL (2025-09-07 10:36:38 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Sep 7 10:01:12.282591417 (20h32m48.322164577s) [upload=18098]: Starting upload Sep 7 10:01:12.282665129 (20h32m48.322238249s) [upload=18098]: Done starting upload Sep 7 10:01:12.282748708 (20h32m48.322321829s) [upload=18098]: Uploading segments... Sep 7 10:01:12.282962778 (20h32m48.322535898s) [upload=18098]: Got next segment. Inline: false Sep 7 10:01:12.282998316 (20h32m48.322571436s) [upload=18098]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 7 10:01:12.283003185 (20h32m48.322576305s) [upload=18098]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 7 10:01:12.554246002 (20h32m48.593819122s) [seg_pos={0 0}, upload=18098]: Begin upload segment... Sep 7 10:01:12.554288213 (20h32m48.593861343s) [seg_pos={0 0}, upload=18098]: Done begin upload segment. Sep 7 10:01:12.554312018 (20h32m48.593885138s) [noise=true , address=45.91.116.12:4225 , piece_id=24NOULLM6NIKWXYDRUSDNF4ZSNZPDX6GGEATNAGLHFS32STQSFXQ, seg_pos={0 0}, upload=18098]: Uploading piece... Sep 7 10:01:12.554351274 (20h32m48.593924394s) [noise=true , address=45.91.116.8:4045 , piece_id=IYSGEW2LSSMHEGU6PSZWW7Q6DKHQQ4752FWRLHAFWRVDAD266T3Q, seg_pos={0 0}, upload=18098]: Uploading piece... Sep 7 10:01:12.554673811 (20h32m48.594246931s) [noise=true , address=77.90.31.18:4805 , piece_id=XSCLLZAWCOYBF37OZQAWQDGG2BXRDLPFZF7Q45YSPKIB6EDFMTMA, seg_pos={0 0}, upload=18098]: Uploading piece... Sep 7 10:01:12.554765747 (20h32m48.594338867s) [noise=true , address=77.90.31.16:4734 , piece_id=EED3HXQB5DTNMGCF45PYPTSGFDJ7BBE4TLEYVANGIVJKPZBDJXSA, seg_pos={0 0}, upload=18098]: Uploading piece... Sep 7 10:01:12.554935050 (20h32m48.59450817s) [noise=true , address=45.91.116.13:4155 , piece_id=TUEYWZN3DX6LMMINAUNUR2YMKOXC2G3TZ2VSJS27VAT54KJIWOVQ, seg_pos={0 0}, upload=18098]: Uploading piece... Sep 7 10:01:12.555030393 (20h32m48.594603513s) [noise=true , address=77.90.31.10:4520 , piece_id=L45DMYCCMRW3Z6WNFDHA3B347VLU3KNHUOI25TXWKUZXJSNZHAEQ, seg_pos={0 0}, upload=18098]: Uploading piece... Sep 7 10:01:12.555205167 (20h32m48.594778297s) [noise=true , address=45.91.116.4:4939 , piece_id=A7DD37CROE5HDXDGUQ3TY6C6FGQR3M6FY3M4X7GO45I6FBWROS2Q, seg_pos={0 0}, upload=18098]: Uploading piece... Sep 7 10:01:12.555359233 (20h32m48.594932353s) [noise=true , address=77.90.31.15:4699 , piece_id=IGGRTGGDMP66HEJMVPVEKJJPI4KEFC4XUPHBOELDIEJSWRZ4WGBQ, seg_pos={0 0}, upload=18098]: Uploading piece... Sep 7 10:01:12.555428104 (20h32m48.595001224s) [noise=true , address=45.91.116.10:4083 , piece_id=TJGO4QHAPO4FOA5VRFN5KLSW54565U6J3AE4I23WKHG4FRPFR5GA, seg_pos={0 0}, upload=18098]: Uploading piece... Sep 7 10:01:12.555587289 (20h32m48.595160419s) [noise=true , address=77.90.31.19:4841 , piece_id=YCDQ2IPAZILCE6EZE3HEKQNATD33FTQMC6SGJ63ZYOJ3HUC7VFPQ, seg_pos={0 0}, upload=18098]: Uploading piece... Sep 7 10:01:12.555665148 (20h32m48.595238268s) [noise=true , address=45.91.116.7:4011 , piece_id=AVIKSOU2RAJAMTQ3DCQZOQ5ZQQC3Y32SOVYAJMGNHC4QTSMNX55Q, seg_pos={0 0}, upload=18098]: Uploading piece... Sep 7 10:01:12.555961315 (20h32m48.595534435s) [noise=true , address=77.90.31.14:4661 , piece_id=BERKPMDU43TSCVU6W36XW3ZMRSETDIWWZUDWLOUJMD54N5I5R6UA, seg_pos={0 0}, upload=18098]: Uploading piece... Sep 7 10:01:12.555973077 (20h32m48.595546207s) [noise=true , address=77.90.31.12:4592 , piece_id=Y7KQVZZL6VRDVNPXH57UPPVIKNPQBLRCVY7PYUV362QZ6CIGN44A, seg_pos={0 0}, upload=18098]: Uploading piece... Sep 7 10:01:12.556368644 (20h32m48.595941764s) [noise=true , address=77.90.31.12:4590 , piece_id=5VLTXP25Y6JPOHBFKEN57P25J3QEL3F54G6Z7UZQTZDRSQKLC27Q, seg_pos={0 0}, upload=18098]: Uploading piece... Sep 7 10:01:12.556645804 (20h32m48.596218924s) [noise=true , address=77.90.31.19:4842 , piece_id=F6IJY4FN7RI6AIK6AW4E62YPS4A36UMZLMELMQLTPJ7ZZS4UXHCQ, seg_pos={0 0}, upload=18098]: Uploading piece... Sep 7 10:01:12.556871486 (20h32m48.596444606s) [noise=true , address=77.90.31.10:4519 , piece_id=5WDKJ2LFXDDD52BHHPLHFEZKKDSQUINZ5UIICKWXMFZYNY7AFKWA, seg_pos={0 0}, upload=18098]: Uploading piece... Sep 7 10:01:14.384175066 (20h32m50.423748186s) [upload=18098]: Next returned nil segment Sep 7 10:01:14.384226745 (20h32m50.423799875s) [upload=18098]: Waiting for error group managing segments... Sep 7 10:01:14.678022165 (20h32m50.717595295s) [noise=true , address=77.90.31.15:4699 , piece_id=IGGRTGGDMP66HEJMVPVEKJJPI4KEFC4XUPHBOELDIEJSWRZ4WGBQ, seg_pos={0 0}, upload=18098]: Done uploading piece. err: <nil> Sep 7 10:01:14.701635289 (20h32m50.74120841s) [noise=true , address=77.90.31.10:4519 , piece_id=5WDKJ2LFXDDD52BHHPLHFEZKKDSQUINZ5UIICKWXMFZYNY7AFKWA, seg_pos={0 0}, upload=18098]: Done uploading piece. err: <nil> Sep 7 10:01:14.722546303 (20h32m50.762119423s) [noise=true , address=45.91.116.4:4939 , piece_id=A7DD37CROE5HDXDGUQ3TY6C6FGQR3M6FY3M4X7GO45I6FBWROS2Q, seg_pos={0 0}, upload=18098]: Done uploading piece. err: <nil> Sep 7 10:01:14.739108335 (20h32m50.778681455s) [noise=true , address=77.90.31.19:4842 , piece_id=F6IJY4FN7RI6AIK6AW4E62YPS4A36UMZLMELMQLTPJ7ZZS4UXHCQ, seg_pos={0 0}, upload=18098]: Done uploading piece. err: <nil> Sep 7 10:01:14.749560815 (20h32m50.789133935s) [noise=true , address=45.91.116.13:4155 , piece_id=TUEYWZN3DX6LMMINAUNUR2YMKOXC2G3TZ2VSJS27VAT54KJIWOVQ, seg_pos={0 0}, upload=18098]: Done uploading piece. err: <nil> Sep 7 10:01:14.754835469 (20h32m50.794408589s) [noise=true , address=77.90.31.19:4841 , piece_id=YCDQ2IPAZILCE6EZE3HEKQNATD33FTQMC6SGJ63ZYOJ3HUC7VFPQ, seg_pos={0 0}, upload=18098]: Done uploading piece. err: <nil> Sep 7 10:01:14.771610288 (20h32m50.811183418s) [noise=true , address=45.91.116.8:4045 , piece_id=IYSGEW2LSSMHEGU6PSZWW7Q6DKHQQ4752FWRLHAFWRVDAD266T3Q, seg_pos={0 0}, upload=18098]: Done uploading piece. err: <nil> Sep 7 10:01:14.798344604 (20h32m50.837917724s) [noise=true , address=77.90.31.16:4734 , piece_id=EED3HXQB5DTNMGCF45PYPTSGFDJ7BBE4TLEYVANGIVJKPZBDJXSA, seg_pos={0 0}, upload=18098]: Done uploading piece. err: <nil> Sep 7 10:01:14.811486902 (20h32m50.851060022s) [noise=true , address=45.91.116.10:4083 , piece_id=TJGO4QHAPO4FOA5VRFN5KLSW54565U6J3AE4I23WKHG4FRPFR5GA, seg_pos={0 0}, upload=18098]: Done uploading piece. err: <nil> Sep 7 10:01:14.822484987 (20h32m50.862058107s) [noise=true , address=77.90.31.14:4661 , piece_id=BERKPMDU43TSCVU6W36XW3ZMRSETDIWWZUDWLOUJMD54N5I5R6UA, seg_pos={0 0}, upload=18098]: Done uploading piece. err: <nil> Sep 7 10:01:14.824396044 (20h32m50.863969164s) [noise=true , address=77.90.31.12:4590 , piece_id=5VLTXP25Y6JPOHBFKEN57P25J3QEL3F54G6Z7UZQTZDRSQKLC27Q, seg_pos={0 0}, upload=18098]: Done uploading piece. err: <nil> Sep 7 10:01:14.843161772 (20h32m50.882734892s) [noise=true , address=45.91.116.7:4011 , piece_id=AVIKSOU2RAJAMTQ3DCQZOQ5ZQQC3Y32SOVYAJMGNHC4QTSMNX55Q, seg_pos={0 0}, upload=18098]: Done uploading piece. err: <nil> Sep 7 10:01:14.849635850 (20h32m50.88920896s) [noise=true , address=45.91.116.12:4225 , piece_id=24NOULLM6NIKWXYDRUSDNF4ZSNZPDX6GGEATNAGLHFS32STQSFXQ, seg_pos={0 0}, upload=18098]: Done uploading piece. err: <nil> Sep 7 10:01:14.849660828 (20h32m50.889233948s) [seg_pos={0 0}, upload=18098]: Segment reached optimal threshold of 13 pieces. Sep 7 10:01:14.849916888 (20h32m50.889490008s) [noise=true , address=77.90.31.12:4592 , piece_id=Y7KQVZZL6VRDVNPXH57UPPVIKNPQBLRCVY7PYUV362QZ6CIGN44A, seg_pos={0 0}, upload=18098]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3): protocol: expected piece hash; context canceled; EOF Sep 7 10:01:14.849984107 (20h32m50.889557227s) [noise=true , address=77.90.31.10:4520 , piece_id=L45DMYCCMRW3Z6WNFDHA3B347VLU3KNHUOI25TXWKUZXJSNZHAEQ, seg_pos={0 0}, upload=18098]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp): protocol: write tcp 188.166.225.170:42490->77.90.31.10:4520: use of closed network connection; write tcp 188.166.225.170:42490->77.90.31.10:4520: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42490->77.90.31.10:4520: use of closed network connection Sep 7 10:01:14.850004236 (20h32m50.889577356s) [noise=true , address=77.90.31.18:4805 , piece_id=XSCLLZAWCOYBF37OZQAWQDGG2BXRDLPFZF7Q45YSPKIB6EDFMTMA, seg_pos={0 0}, upload=18098]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT): protocol: write tcp 188.166.225.170:52276->77.90.31.18:4805: use of closed network connection; write tcp 188.166.225.170:52276->77.90.31.18:4805: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52276->77.90.31.18:4805: use of closed network connection Sep 7 10:01:14.850018062 (20h32m50.889591182s) [seg_pos={0 0}, upload=18098]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 7 10:01:14.850083738 (20h32m50.889656858s) [upload=18098]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 7 10:01:14.850087495 (20h32m50.889660615s) [upload=18098]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 7 10:01:15.305723891 (20h32m51.345297021s) [upload=18098]: Done uploading segments... Sep 7 10:01:15.305740914 (20h32m51.345314034s) [upload=18098]: Upload finished. err: <nil> Duration: 7.260958642s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.....X.....X................................................] duration: 6.398520183s FAIL (2025-09-07 10:36:16 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Sep 7 10:01:12.282591417 (20h32m48.322164577s) [upload=18098]: Starting upload Sep 7 10:01:12.282665129 (20h32m48.322238249s) [upload=18098]: Done starting upload Sep 7 10:01:12.282748708 (20h32m48.322321829s) [upload=18098]: Uploading segments... Sep 7 10:01:12.282962778 (20h32m48.322535898s) [upload=18098]: Got next segment. Inline: false Sep 7 10:01:12.282998316 (20h32m48.322571436s) [upload=18098]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 7 10:01:12.283003185 (20h32m48.322576305s) [upload=18098]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 7 10:01:12.554246002 (20h32m48.593819122s) [seg_pos={0 0}, upload=18098]: Begin upload segment... Sep 7 10:01:12.554288213 (20h32m48.593861343s) [seg_pos={0 0}, upload=18098]: Done begin upload segment. Sep 7 10:01:12.554312018 (20h32m48.593885138s) [noise=true , address=45.91.116.12:4225 , piece_id=24NOULLM6NIKWXYDRUSDNF4ZSNZPDX6GGEATNAGLHFS32STQSFXQ, seg_pos={0 0}, upload=18098]: Uploading piece... Sep 7 10:01:12.554351274 (20h32m48.593924394s) [noise=true , address=45.91.116.8:4045 , piece_id=IYSGEW2LSSMHEGU6PSZWW7Q6DKHQQ4752FWRLHAFWRVDAD266T3Q, seg_pos={0 0}, upload=18098]: Uploading piece... Sep 7 10:01:12.554673811 (20h32m48.594246931s) [noise=true , address=77.90.31.18:4805 , piece_id=XSCLLZAWCOYBF37OZQAWQDGG2BXRDLPFZF7Q45YSPKIB6EDFMTMA, seg_pos={0 0}, upload=18098]: Uploading piece... Sep 7 10:01:12.554765747 (20h32m48.594338867s) [noise=true , address=77.90.31.16:4734 , piece_id=EED3HXQB5DTNMGCF45PYPTSGFDJ7BBE4TLEYVANGIVJKPZBDJXSA, seg_pos={0 0}, upload=18098]: Uploading piece... Sep 7 10:01:12.554935050 (20h32m48.59450817s) [noise=true , address=45.91.116.13:4155 , piece_id=TUEYWZN3DX6LMMINAUNUR2YMKOXC2G3TZ2VSJS27VAT54KJIWOVQ, seg_pos={0 0}, upload=18098]: Uploading piece... Sep 7 10:01:12.555030393 (20h32m48.594603513s) [noise=true , address=77.90.31.10:4520 , piece_id=L45DMYCCMRW3Z6WNFDHA3B347VLU3KNHUOI25TXWKUZXJSNZHAEQ, seg_pos={0 0}, upload=18098]: Uploading piece... Sep 7 10:01:12.555205167 (20h32m48.594778297s) [noise=true , address=45.91.116.4:4939 , piece_id=A7DD37CROE5HDXDGUQ3TY6C6FGQR3M6FY3M4X7GO45I6FBWROS2Q, seg_pos={0 0}, upload=18098]: Uploading piece... Sep 7 10:01:12.555359233 (20h32m48.594932353s) [noise=true , address=77.90.31.15:4699 , piece_id=IGGRTGGDMP66HEJMVPVEKJJPI4KEFC4XUPHBOELDIEJSWRZ4WGBQ, seg_pos={0 0}, upload=18098]: Uploading piece... Sep 7 10:01:12.555428104 (20h32m48.595001224s) [noise=true , address=45.91.116.10:4083 , piece_id=TJGO4QHAPO4FOA5VRFN5KLSW54565U6J3AE4I23WKHG4FRPFR5GA, seg_pos={0 0}, upload=18098]: Uploading piece... Sep 7 10:01:12.555587289 (20h32m48.595160419s) [noise=true , address=77.90.31.19:4841 , piece_id=YCDQ2IPAZILCE6EZE3HEKQNATD33FTQMC6SGJ63ZYOJ3HUC7VFPQ, seg_pos={0 0}, upload=18098]: Uploading piece... Sep 7 10:01:12.555665148 (20h32m48.595238268s) [noise=true , address=45.91.116.7:4011 , piece_id=AVIKSOU2RAJAMTQ3DCQZOQ5ZQQC3Y32SOVYAJMGNHC4QTSMNX55Q, seg_pos={0 0}, upload=18098]: Uploading piece... Sep 7 10:01:12.555961315 (20h32m48.595534435s) [noise=true , address=77.90.31.14:4661 , piece_id=BERKPMDU43TSCVU6W36XW3ZMRSETDIWWZUDWLOUJMD54N5I5R6UA, seg_pos={0 0}, upload=18098]: Uploading piece... Sep 7 10:01:12.555973077 (20h32m48.595546207s) [noise=true , address=77.90.31.12:4592 , piece_id=Y7KQVZZL6VRDVNPXH57UPPVIKNPQBLRCVY7PYUV362QZ6CIGN44A, seg_pos={0 0}, upload=18098]: Uploading piece... Sep 7 10:01:12.556368644 (20h32m48.595941764s) [noise=true , address=77.90.31.12:4590 , piece_id=5VLTXP25Y6JPOHBFKEN57P25J3QEL3F54G6Z7UZQTZDRSQKLC27Q, seg_pos={0 0}, upload=18098]: Uploading piece... Sep 7 10:01:12.556645804 (20h32m48.596218924s) [noise=true , address=77.90.31.19:4842 , piece_id=F6IJY4FN7RI6AIK6AW4E62YPS4A36UMZLMELMQLTPJ7ZZS4UXHCQ, seg_pos={0 0}, upload=18098]: Uploading piece... Sep 7 10:01:12.556871486 (20h32m48.596444606s) [noise=true , address=77.90.31.10:4519 , piece_id=5WDKJ2LFXDDD52BHHPLHFEZKKDSQUINZ5UIICKWXMFZYNY7AFKWA, seg_pos={0 0}, upload=18098]: Uploading piece... Sep 7 10:01:14.384175066 (20h32m50.423748186s) [upload=18098]: Next returned nil segment Sep 7 10:01:14.384226745 (20h32m50.423799875s) [upload=18098]: Waiting for error group managing segments... Sep 7 10:01:14.678022165 (20h32m50.717595295s) [noise=true , address=77.90.31.15:4699 , piece_id=IGGRTGGDMP66HEJMVPVEKJJPI4KEFC4XUPHBOELDIEJSWRZ4WGBQ, seg_pos={0 0}, upload=18098]: Done uploading piece. err: <nil> Sep 7 10:01:14.701635289 (20h32m50.74120841s) [noise=true , address=77.90.31.10:4519 , piece_id=5WDKJ2LFXDDD52BHHPLHFEZKKDSQUINZ5UIICKWXMFZYNY7AFKWA, seg_pos={0 0}, upload=18098]: Done uploading piece. err: <nil> Sep 7 10:01:14.722546303 (20h32m50.762119423s) [noise=true , address=45.91.116.4:4939 , piece_id=A7DD37CROE5HDXDGUQ3TY6C6FGQR3M6FY3M4X7GO45I6FBWROS2Q, seg_pos={0 0}, upload=18098]: Done uploading piece. err: <nil> Sep 7 10:01:14.739108335 (20h32m50.778681455s) [noise=true , address=77.90.31.19:4842 , piece_id=F6IJY4FN7RI6AIK6AW4E62YPS4A36UMZLMELMQLTPJ7ZZS4UXHCQ, seg_pos={0 0}, upload=18098]: Done uploading piece. err: <nil> Sep 7 10:01:14.749560815 (20h32m50.789133935s) [noise=true , address=45.91.116.13:4155 , piece_id=TUEYWZN3DX6LMMINAUNUR2YMKOXC2G3TZ2VSJS27VAT54KJIWOVQ, seg_pos={0 0}, upload=18098]: Done uploading piece. err: <nil> Sep 7 10:01:14.754835469 (20h32m50.794408589s) [noise=true , address=77.90.31.19:4841 , piece_id=YCDQ2IPAZILCE6EZE3HEKQNATD33FTQMC6SGJ63ZYOJ3HUC7VFPQ, seg_pos={0 0}, upload=18098]: Done uploading piece. err: <nil> Sep 7 10:01:14.771610288 (20h32m50.811183418s) [noise=true , address=45.91.116.8:4045 , piece_id=IYSGEW2LSSMHEGU6PSZWW7Q6DKHQQ4752FWRLHAFWRVDAD266T3Q, seg_pos={0 0}, upload=18098]: Done uploading piece. err: <nil> Sep 7 10:01:14.798344604 (20h32m50.837917724s) [noise=true , address=77.90.31.16:4734 , piece_id=EED3HXQB5DTNMGCF45PYPTSGFDJ7BBE4TLEYVANGIVJKPZBDJXSA, seg_pos={0 0}, upload=18098]: Done uploading piece. err: <nil> Sep 7 10:01:14.811486902 (20h32m50.851060022s) [noise=true , address=45.91.116.10:4083 , piece_id=TJGO4QHAPO4FOA5VRFN5KLSW54565U6J3AE4I23WKHG4FRPFR5GA, seg_pos={0 0}, upload=18098]: Done uploading piece. err: <nil> Sep 7 10:01:14.822484987 (20h32m50.862058107s) [noise=true , address=77.90.31.14:4661 , piece_id=BERKPMDU43TSCVU6W36XW3ZMRSETDIWWZUDWLOUJMD54N5I5R6UA, seg_pos={0 0}, upload=18098]: Done uploading piece. err: <nil> Sep 7 10:01:14.824396044 (20h32m50.863969164s) [noise=true , address=77.90.31.12:4590 , piece_id=5VLTXP25Y6JPOHBFKEN57P25J3QEL3F54G6Z7UZQTZDRSQKLC27Q, seg_pos={0 0}, upload=18098]: Done uploading piece. err: <nil> Sep 7 10:01:14.843161772 (20h32m50.882734892s) [noise=true , address=45.91.116.7:4011 , piece_id=AVIKSOU2RAJAMTQ3DCQZOQ5ZQQC3Y32SOVYAJMGNHC4QTSMNX55Q, seg_pos={0 0}, upload=18098]: Done uploading piece. err: <nil> Sep 7 10:01:14.849635850 (20h32m50.88920896s) [noise=true , address=45.91.116.12:4225 , piece_id=24NOULLM6NIKWXYDRUSDNF4ZSNZPDX6GGEATNAGLHFS32STQSFXQ, seg_pos={0 0}, upload=18098]: Done uploading piece. err: <nil> Sep 7 10:01:14.849660828 (20h32m50.889233948s) [seg_pos={0 0}, upload=18098]: Segment reached optimal threshold of 13 pieces. Sep 7 10:01:14.849916888 (20h32m50.889490008s) [noise=true , address=77.90.31.12:4592 , piece_id=Y7KQVZZL6VRDVNPXH57UPPVIKNPQBLRCVY7PYUV362QZ6CIGN44A, seg_pos={0 0}, upload=18098]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3): protocol: expected piece hash; context canceled; EOF Sep 7 10:01:14.849984107 (20h32m50.889557227s) [noise=true , address=77.90.31.10:4520 , piece_id=L45DMYCCMRW3Z6WNFDHA3B347VLU3KNHUOI25TXWKUZXJSNZHAEQ, seg_pos={0 0}, upload=18098]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp): protocol: write tcp 188.166.225.170:42490->77.90.31.10:4520: use of closed network connection; write tcp 188.166.225.170:42490->77.90.31.10:4520: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42490->77.90.31.10:4520: use of closed network connection Sep 7 10:01:14.850004236 (20h32m50.889577356s) [noise=true , address=77.90.31.18:4805 , piece_id=XSCLLZAWCOYBF37OZQAWQDGG2BXRDLPFZF7Q45YSPKIB6EDFMTMA, seg_pos={0 0}, upload=18098]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT): protocol: write tcp 188.166.225.170:52276->77.90.31.18:4805: use of closed network connection; write tcp 188.166.225.170:52276->77.90.31.18:4805: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52276->77.90.31.18:4805: use of closed network connection Sep 7 10:01:14.850018062 (20h32m50.889591182s) [seg_pos={0 0}, upload=18098]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 7 10:01:14.850083738 (20h32m50.889656858s) [upload=18098]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 7 10:01:14.850087495 (20h32m50.889660615s) [upload=18098]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 7 10:01:15.305723891 (20h32m51.345297021s) [upload=18098]: Done uploading segments... Sep 7 10:01:15.305740914 (20h32m51.345314034s) [upload=18098]: Upload finished. err: <nil> Duration: 6.398555101s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [......X.....X...............................................] duration: 11.418369853s FAIL (2025-09-07 10:35:06 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Sep 7 10:01:12.282591417 (20h32m48.322164577s) [upload=18098]: Starting upload Sep 7 10:01:12.282665129 (20h32m48.322238249s) [upload=18098]: Done starting upload Sep 7 10:01:12.282748708 (20h32m48.322321829s) [upload=18098]: Uploading segments... Sep 7 10:01:12.282962778 (20h32m48.322535898s) [upload=18098]: Got next segment. Inline: false Sep 7 10:01:12.282998316 (20h32m48.322571436s) [upload=18098]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 7 10:01:12.283003185 (20h32m48.322576305s) [upload=18098]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 7 10:01:12.554246002 (20h32m48.593819122s) [seg_pos={0 0}, upload=18098]: Begin upload segment... Sep 7 10:01:12.554288213 (20h32m48.593861343s) [seg_pos={0 0}, upload=18098]: Done begin upload segment. Sep 7 10:01:12.554312018 (20h32m48.593885138s) [noise=true , address=45.91.116.12:4225 , piece_id=24NOULLM6NIKWXYDRUSDNF4ZSNZPDX6GGEATNAGLHFS32STQSFXQ, seg_pos={0 0}, upload=18098]: Uploading piece... Sep 7 10:01:12.554351274 (20h32m48.593924394s) [noise=true , address=45.91.116.8:4045 , piece_id=IYSGEW2LSSMHEGU6PSZWW7Q6DKHQQ4752FWRLHAFWRVDAD266T3Q, seg_pos={0 0}, upload=18098]: Uploading piece... Sep 7 10:01:12.554673811 (20h32m48.594246931s) [noise=true , address=77.90.31.18:4805 , piece_id=XSCLLZAWCOYBF37OZQAWQDGG2BXRDLPFZF7Q45YSPKIB6EDFMTMA, seg_pos={0 0}, upload=18098]: Uploading piece... Sep 7 10:01:12.554765747 (20h32m48.594338867s) [noise=true , address=77.90.31.16:4734 , piece_id=EED3HXQB5DTNMGCF45PYPTSGFDJ7BBE4TLEYVANGIVJKPZBDJXSA, seg_pos={0 0}, upload=18098]: Uploading piece... Sep 7 10:01:12.554935050 (20h32m48.59450817s) [noise=true , address=45.91.116.13:4155 , piece_id=TUEYWZN3DX6LMMINAUNUR2YMKOXC2G3TZ2VSJS27VAT54KJIWOVQ, seg_pos={0 0}, upload=18098]: Uploading piece... Sep 7 10:01:12.555030393 (20h32m48.594603513s) [noise=true , address=77.90.31.10:4520 , piece_id=L45DMYCCMRW3Z6WNFDHA3B347VLU3KNHUOI25TXWKUZXJSNZHAEQ, seg_pos={0 0}, upload=18098]: Uploading piece... Sep 7 10:01:12.555205167 (20h32m48.594778297s) [noise=true , address=45.91.116.4:4939 , piece_id=A7DD37CROE5HDXDGUQ3TY6C6FGQR3M6FY3M4X7GO45I6FBWROS2Q, seg_pos={0 0}, upload=18098]: Uploading piece... Sep 7 10:01:12.555359233 (20h32m48.594932353s) [noise=true , address=77.90.31.15:4699 , piece_id=IGGRTGGDMP66HEJMVPVEKJJPI4KEFC4XUPHBOELDIEJSWRZ4WGBQ, seg_pos={0 0}, upload=18098]: Uploading piece... Sep 7 10:01:12.555428104 (20h32m48.595001224s) [noise=true , address=45.91.116.10:4083 , piece_id=TJGO4QHAPO4FOA5VRFN5KLSW54565U6J3AE4I23WKHG4FRPFR5GA, seg_pos={0 0}, upload=18098]: Uploading piece... Sep 7 10:01:12.555587289 (20h32m48.595160419s) [noise=true , address=77.90.31.19:4841 , piece_id=YCDQ2IPAZILCE6EZE3HEKQNATD33FTQMC6SGJ63ZYOJ3HUC7VFPQ, seg_pos={0 0}, upload=18098]: Uploading piece... Sep 7 10:01:12.555665148 (20h32m48.595238268s) [noise=true , address=45.91.116.7:4011 , piece_id=AVIKSOU2RAJAMTQ3DCQZOQ5ZQQC3Y32SOVYAJMGNHC4QTSMNX55Q, seg_pos={0 0}, upload=18098]: Uploading piece... Sep 7 10:01:12.555961315 (20h32m48.595534435s) [noise=true , address=77.90.31.14:4661 , piece_id=BERKPMDU43TSCVU6W36XW3ZMRSETDIWWZUDWLOUJMD54N5I5R6UA, seg_pos={0 0}, upload=18098]: Uploading piece... Sep 7 10:01:12.555973077 (20h32m48.595546207s) [noise=true , address=77.90.31.12:4592 , piece_id=Y7KQVZZL6VRDVNPXH57UPPVIKNPQBLRCVY7PYUV362QZ6CIGN44A, seg_pos={0 0}, upload=18098]: Uploading piece... Sep 7 10:01:12.556368644 (20h32m48.595941764s) [noise=true , address=77.90.31.12:4590 , piece_id=5VLTXP25Y6JPOHBFKEN57P25J3QEL3F54G6Z7UZQTZDRSQKLC27Q, seg_pos={0 0}, upload=18098]: Uploading piece... Sep 7 10:01:12.556645804 (20h32m48.596218924s) [noise=true , address=77.90.31.19:4842 , piece_id=F6IJY4FN7RI6AIK6AW4E62YPS4A36UMZLMELMQLTPJ7ZZS4UXHCQ, seg_pos={0 0}, upload=18098]: Uploading piece... Sep 7 10:01:12.556871486 (20h32m48.596444606s) [noise=true , address=77.90.31.10:4519 , piece_id=5WDKJ2LFXDDD52BHHPLHFEZKKDSQUINZ5UIICKWXMFZYNY7AFKWA, seg_pos={0 0}, upload=18098]: Uploading piece... Sep 7 10:01:14.384175066 (20h32m50.423748186s) [upload=18098]: Next returned nil segment Sep 7 10:01:14.384226745 (20h32m50.423799875s) [upload=18098]: Waiting for error group managing segments... Sep 7 10:01:14.678022165 (20h32m50.717595295s) [noise=true , address=77.90.31.15:4699 , piece_id=IGGRTGGDMP66HEJMVPVEKJJPI4KEFC4XUPHBOELDIEJSWRZ4WGBQ, seg_pos={0 0}, upload=18098]: Done uploading piece. err: <nil> Sep 7 10:01:14.701635289 (20h32m50.74120841s) [noise=true , address=77.90.31.10:4519 , piece_id=5WDKJ2LFXDDD52BHHPLHFEZKKDSQUINZ5UIICKWXMFZYNY7AFKWA, seg_pos={0 0}, upload=18098]: Done uploading piece. err: <nil> Sep 7 10:01:14.722546303 (20h32m50.762119423s) [noise=true , address=45.91.116.4:4939 , piece_id=A7DD37CROE5HDXDGUQ3TY6C6FGQR3M6FY3M4X7GO45I6FBWROS2Q, seg_pos={0 0}, upload=18098]: Done uploading piece. err: <nil> Sep 7 10:01:14.739108335 (20h32m50.778681455s) [noise=true , address=77.90.31.19:4842 , piece_id=F6IJY4FN7RI6AIK6AW4E62YPS4A36UMZLMELMQLTPJ7ZZS4UXHCQ, seg_pos={0 0}, upload=18098]: Done uploading piece. err: <nil> Sep 7 10:01:14.749560815 (20h32m50.789133935s) [noise=true , address=45.91.116.13:4155 , piece_id=TUEYWZN3DX6LMMINAUNUR2YMKOXC2G3TZ2VSJS27VAT54KJIWOVQ, seg_pos={0 0}, upload=18098]: Done uploading piece. err: <nil> Sep 7 10:01:14.754835469 (20h32m50.794408589s) [noise=true , address=77.90.31.19:4841 , piece_id=YCDQ2IPAZILCE6EZE3HEKQNATD33FTQMC6SGJ63ZYOJ3HUC7VFPQ, seg_pos={0 0}, upload=18098]: Done uploading piece. err: <nil> Sep 7 10:01:14.771610288 (20h32m50.811183418s) [noise=true , address=45.91.116.8:4045 , piece_id=IYSGEW2LSSMHEGU6PSZWW7Q6DKHQQ4752FWRLHAFWRVDAD266T3Q, seg_pos={0 0}, upload=18098]: Done uploading piece. err: <nil> Sep 7 10:01:14.798344604 (20h32m50.837917724s) [noise=true , address=77.90.31.16:4734 , piece_id=EED3HXQB5DTNMGCF45PYPTSGFDJ7BBE4TLEYVANGIVJKPZBDJXSA, seg_pos={0 0}, upload=18098]: Done uploading piece. err: <nil> Sep 7 10:01:14.811486902 (20h32m50.851060022s) [noise=true , address=45.91.116.10:4083 , piece_id=TJGO4QHAPO4FOA5VRFN5KLSW54565U6J3AE4I23WKHG4FRPFR5GA, seg_pos={0 0}, upload=18098]: Done uploading piece. err: <nil> Sep 7 10:01:14.822484987 (20h32m50.862058107s) [noise=true , address=77.90.31.14:4661 , piece_id=BERKPMDU43TSCVU6W36XW3ZMRSETDIWWZUDWLOUJMD54N5I5R6UA, seg_pos={0 0}, upload=18098]: Done uploading piece. err: <nil> Sep 7 10:01:14.824396044 (20h32m50.863969164s) [noise=true , address=77.90.31.12:4590 , piece_id=5VLTXP25Y6JPOHBFKEN57P25J3QEL3F54G6Z7UZQTZDRSQKLC27Q, seg_pos={0 0}, upload=18098]: Done uploading piece. err: <nil> Sep 7 10:01:14.843161772 (20h32m50.882734892s) [noise=true , address=45.91.116.7:4011 , piece_id=AVIKSOU2RAJAMTQ3DCQZOQ5ZQQC3Y32SOVYAJMGNHC4QTSMNX55Q, seg_pos={0 0}, upload=18098]: Done uploading piece. err: <nil> Sep 7 10:01:14.849635850 (20h32m50.88920896s) [noise=true , address=45.91.116.12:4225 , piece_id=24NOULLM6NIKWXYDRUSDNF4ZSNZPDX6GGEATNAGLHFS32STQSFXQ, seg_pos={0 0}, upload=18098]: Done uploading piece. err: <nil> Sep 7 10:01:14.849660828 (20h32m50.889233948s) [seg_pos={0 0}, upload=18098]: Segment reached optimal threshold of 13 pieces. Sep 7 10:01:14.849916888 (20h32m50.889490008s) [noise=true , address=77.90.31.12:4592 , piece_id=Y7KQVZZL6VRDVNPXH57UPPVIKNPQBLRCVY7PYUV362QZ6CIGN44A, seg_pos={0 0}, upload=18098]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3): protocol: expected piece hash; context canceled; EOF Sep 7 10:01:14.849984107 (20h32m50.889557227s) [noise=true , address=77.90.31.10:4520 , piece_id=L45DMYCCMRW3Z6WNFDHA3B347VLU3KNHUOI25TXWKUZXJSNZHAEQ, seg_pos={0 0}, upload=18098]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp): protocol: write tcp 188.166.225.170:42490->77.90.31.10:4520: use of closed network connection; write tcp 188.166.225.170:42490->77.90.31.10:4520: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42490->77.90.31.10:4520: use of closed network connection Sep 7 10:01:14.850004236 (20h32m50.889577356s) [noise=true , address=77.90.31.18:4805 , piece_id=XSCLLZAWCOYBF37OZQAWQDGG2BXRDLPFZF7Q45YSPKIB6EDFMTMA, seg_pos={0 0}, upload=18098]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT): protocol: write tcp 188.166.225.170:52276->77.90.31.18:4805: use of closed network connection; write tcp 188.166.225.170:52276->77.90.31.18:4805: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52276->77.90.31.18:4805: use of closed network connection Sep 7 10:01:14.850018062 (20h32m50.889591182s) [seg_pos={0 0}, upload=18098]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 7 10:01:14.850083738 (20h32m50.889656858s) [upload=18098]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 7 10:01:14.850087495 (20h32m50.889660615s) [upload=18098]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 7 10:01:15.305723891 (20h32m51.345297021s) [upload=18098]: Done uploading segments... Sep 7 10:01:15.305740914 (20h32m51.345314034s) [upload=18098]: Upload finished. err: <nil> Duration: 11.418403327s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.......X.....X..............................................] duration: 6.308294277s FAIL (2025-09-07 10:34:46 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Sep 7 10:01:12.282591417 (20h32m48.322164577s) [upload=18098]: Starting upload Sep 7 10:01:12.282665129 (20h32m48.322238249s) [upload=18098]: Done starting upload Sep 7 10:01:12.282748708 (20h32m48.322321829s) [upload=18098]: Uploading segments... Sep 7 10:01:12.282962778 (20h32m48.322535898s) [upload=18098]: Got next segment. Inline: false Sep 7 10:01:12.282998316 (20h32m48.322571436s) [upload=18098]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 7 10:01:12.283003185 (20h32m48.322576305s) [upload=18098]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 7 10:01:12.554246002 (20h32m48.593819122s) [seg_pos={0 0}, upload=18098]: Begin upload segment... Sep 7 10:01:12.554288213 (20h32m48.593861343s) [seg_pos={0 0}, upload=18098]: Done begin upload segment. Sep 7 10:01:12.554312018 (20h32m48.593885138s) [noise=true , address=45.91.116.12:4225 , piece_id=24NOULLM6NIKWXYDRUSDNF4ZSNZPDX6GGEATNAGLHFS32STQSFXQ, seg_pos={0 0}, upload=18098]: Uploading piece... Sep 7 10:01:12.554351274 (20h32m48.593924394s) [noise=true , address=45.91.116.8:4045 , piece_id=IYSGEW2LSSMHEGU6PSZWW7Q6DKHQQ4752FWRLHAFWRVDAD266T3Q, seg_pos={0 0}, upload=18098]: Uploading piece... Sep 7 10:01:12.554673811 (20h32m48.594246931s) [noise=true , address=77.90.31.18:4805 , piece_id=XSCLLZAWCOYBF37OZQAWQDGG2BXRDLPFZF7Q45YSPKIB6EDFMTMA, seg_pos={0 0}, upload=18098]: Uploading piece... Sep 7 10:01:12.554765747 (20h32m48.594338867s) [noise=true , address=77.90.31.16:4734 , piece_id=EED3HXQB5DTNMGCF45PYPTSGFDJ7BBE4TLEYVANGIVJKPZBDJXSA, seg_pos={0 0}, upload=18098]: Uploading piece... Sep 7 10:01:12.554935050 (20h32m48.59450817s) [noise=true , address=45.91.116.13:4155 , piece_id=TUEYWZN3DX6LMMINAUNUR2YMKOXC2G3TZ2VSJS27VAT54KJIWOVQ, seg_pos={0 0}, upload=18098]: Uploading piece... Sep 7 10:01:12.555030393 (20h32m48.594603513s) [noise=true , address=77.90.31.10:4520 , piece_id=L45DMYCCMRW3Z6WNFDHA3B347VLU3KNHUOI25TXWKUZXJSNZHAEQ, seg_pos={0 0}, upload=18098]: Uploading piece... Sep 7 10:01:12.555205167 (20h32m48.594778297s) [noise=true , address=45.91.116.4:4939 , piece_id=A7DD37CROE5HDXDGUQ3TY6C6FGQR3M6FY3M4X7GO45I6FBWROS2Q, seg_pos={0 0}, upload=18098]: Uploading piece... Sep 7 10:01:12.555359233 (20h32m48.594932353s) [noise=true , address=77.90.31.15:4699 , piece_id=IGGRTGGDMP66HEJMVPVEKJJPI4KEFC4XUPHBOELDIEJSWRZ4WGBQ, seg_pos={0 0}, upload=18098]: Uploading piece... Sep 7 10:01:12.555428104 (20h32m48.595001224s) [noise=true , address=45.91.116.10:4083 , piece_id=TJGO4QHAPO4FOA5VRFN5KLSW54565U6J3AE4I23WKHG4FRPFR5GA, seg_pos={0 0}, upload=18098]: Uploading piece... Sep 7 10:01:12.555587289 (20h32m48.595160419s) [noise=true , address=77.90.31.19:4841 , piece_id=YCDQ2IPAZILCE6EZE3HEKQNATD33FTQMC6SGJ63ZYOJ3HUC7VFPQ, seg_pos={0 0}, upload=18098]: Uploading piece... Sep 7 10:01:12.555665148 (20h32m48.595238268s) [noise=true , address=45.91.116.7:4011 , piece_id=AVIKSOU2RAJAMTQ3DCQZOQ5ZQQC3Y32SOVYAJMGNHC4QTSMNX55Q, seg_pos={0 0}, upload=18098]: Uploading piece... Sep 7 10:01:12.555961315 (20h32m48.595534435s) [noise=true , address=77.90.31.14:4661 , piece_id=BERKPMDU43TSCVU6W36XW3ZMRSETDIWWZUDWLOUJMD54N5I5R6UA, seg_pos={0 0}, upload=18098]: Uploading piece... Sep 7 10:01:12.555973077 (20h32m48.595546207s) [noise=true , address=77.90.31.12:4592 , piece_id=Y7KQVZZL6VRDVNPXH57UPPVIKNPQBLRCVY7PYUV362QZ6CIGN44A, seg_pos={0 0}, upload=18098]: Uploading piece... Sep 7 10:01:12.556368644 (20h32m48.595941764s) [noise=true , address=77.90.31.12:4590 , piece_id=5VLTXP25Y6JPOHBFKEN57P25J3QEL3F54G6Z7UZQTZDRSQKLC27Q, seg_pos={0 0}, upload=18098]: Uploading piece... Sep 7 10:01:12.556645804 (20h32m48.596218924s) [noise=true , address=77.90.31.19:4842 , piece_id=F6IJY4FN7RI6AIK6AW4E62YPS4A36UMZLMELMQLTPJ7ZZS4UXHCQ, seg_pos={0 0}, upload=18098]: Uploading piece... Sep 7 10:01:12.556871486 (20h32m48.596444606s) [noise=true , address=77.90.31.10:4519 , piece_id=5WDKJ2LFXDDD52BHHPLHFEZKKDSQUINZ5UIICKWXMFZYNY7AFKWA, seg_pos={0 0}, upload=18098]: Uploading piece... Sep 7 10:01:14.384175066 (20h32m50.423748186s) [upload=18098]: Next returned nil segment Sep 7 10:01:14.384226745 (20h32m50.423799875s) [upload=18098]: Waiting for error group managing segments... Sep 7 10:01:14.678022165 (20h32m50.717595295s) [noise=true , address=77.90.31.15:4699 , piece_id=IGGRTGGDMP66HEJMVPVEKJJPI4KEFC4XUPHBOELDIEJSWRZ4WGBQ, seg_pos={0 0}, upload=18098]: Done uploading piece. err: <nil> Sep 7 10:01:14.701635289 (20h32m50.74120841s) [noise=true , address=77.90.31.10:4519 , piece_id=5WDKJ2LFXDDD52BHHPLHFEZKKDSQUINZ5UIICKWXMFZYNY7AFKWA, seg_pos={0 0}, upload=18098]: Done uploading piece. err: <nil> Sep 7 10:01:14.722546303 (20h32m50.762119423s) [noise=true , address=45.91.116.4:4939 , piece_id=A7DD37CROE5HDXDGUQ3TY6C6FGQR3M6FY3M4X7GO45I6FBWROS2Q, seg_pos={0 0}, upload=18098]: Done uploading piece. err: <nil> Sep 7 10:01:14.739108335 (20h32m50.778681455s) [noise=true , address=77.90.31.19:4842 , piece_id=F6IJY4FN7RI6AIK6AW4E62YPS4A36UMZLMELMQLTPJ7ZZS4UXHCQ, seg_pos={0 0}, upload=18098]: Done uploading piece. err: <nil> Sep 7 10:01:14.749560815 (20h32m50.789133935s) [noise=true , address=45.91.116.13:4155 , piece_id=TUEYWZN3DX6LMMINAUNUR2YMKOXC2G3TZ2VSJS27VAT54KJIWOVQ, seg_pos={0 0}, upload=18098]: Done uploading piece. err: <nil> Sep 7 10:01:14.754835469 (20h32m50.794408589s) [noise=true , address=77.90.31.19:4841 , piece_id=YCDQ2IPAZILCE6EZE3HEKQNATD33FTQMC6SGJ63ZYOJ3HUC7VFPQ, seg_pos={0 0}, upload=18098]: Done uploading piece. err: <nil> Sep 7 10:01:14.771610288 (20h32m50.811183418s) [noise=true , address=45.91.116.8:4045 , piece_id=IYSGEW2LSSMHEGU6PSZWW7Q6DKHQQ4752FWRLHAFWRVDAD266T3Q, seg_pos={0 0}, upload=18098]: Done uploading piece. err: <nil> Sep 7 10:01:14.798344604 (20h32m50.837917724s) [noise=true , address=77.90.31.16:4734 , piece_id=EED3HXQB5DTNMGCF45PYPTSGFDJ7BBE4TLEYVANGIVJKPZBDJXSA, seg_pos={0 0}, upload=18098]: Done uploading piece. err: <nil> Sep 7 10:01:14.811486902 (20h32m50.851060022s) [noise=true , address=45.91.116.10:4083 , piece_id=TJGO4QHAPO4FOA5VRFN5KLSW54565U6J3AE4I23WKHG4FRPFR5GA, seg_pos={0 0}, upload=18098]: Done uploading piece. err: <nil> Sep 7 10:01:14.822484987 (20h32m50.862058107s) [noise=true , address=77.90.31.14:4661 , piece_id=BERKPMDU43TSCVU6W36XW3ZMRSETDIWWZUDWLOUJMD54N5I5R6UA, seg_pos={0 0}, upload=18098]: Done uploading piece. err: <nil> Sep 7 10:01:14.824396044 (20h32m50.863969164s) [noise=true , address=77.90.31.12:4590 , piece_id=5VLTXP25Y6JPOHBFKEN57P25J3QEL3F54G6Z7UZQTZDRSQKLC27Q, seg_pos={0 0}, upload=18098]: Done uploading piece. err: <nil> Sep 7 10:01:14.843161772 (20h32m50.882734892s) [noise=true , address=45.91.116.7:4011 , piece_id=AVIKSOU2RAJAMTQ3DCQZOQ5ZQQC3Y32SOVYAJMGNHC4QTSMNX55Q, seg_pos={0 0}, upload=18098]: Done uploading piece. err: <nil> Sep 7 10:01:14.849635850 (20h32m50.88920896s) [noise=true , address=45.91.116.12:4225 , piece_id=24NOULLM6NIKWXYDRUSDNF4ZSNZPDX6GGEATNAGLHFS32STQSFXQ, seg_pos={0 0}, upload=18098]: Done uploading piece. err: <nil> Sep 7 10:01:14.849660828 (20h32m50.889233948s) [seg_pos={0 0}, upload=18098]: Segment reached optimal threshold of 13 pieces. Sep 7 10:01:14.849916888 (20h32m50.889490008s) [noise=true , address=77.90.31.12:4592 , piece_id=Y7KQVZZL6VRDVNPXH57UPPVIKNPQBLRCVY7PYUV362QZ6CIGN44A, seg_pos={0 0}, upload=18098]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3): protocol: expected piece hash; context canceled; EOF Sep 7 10:01:14.849984107 (20h32m50.889557227s) [noise=true , address=77.90.31.10:4520 , piece_id=L45DMYCCMRW3Z6WNFDHA3B347VLU3KNHUOI25TXWKUZXJSNZHAEQ, seg_pos={0 0}, upload=18098]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp): protocol: write tcp 188.166.225.170:42490->77.90.31.10:4520: use of closed network connection; write tcp 188.166.225.170:42490->77.90.31.10:4520: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42490->77.90.31.10:4520: use of closed network connection Sep 7 10:01:14.850004236 (20h32m50.889577356s) [noise=true , address=77.90.31.18:4805 , piece_id=XSCLLZAWCOYBF37OZQAWQDGG2BXRDLPFZF7Q45YSPKIB6EDFMTMA, seg_pos={0 0}, upload=18098]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT): protocol: write tcp 188.166.225.170:52276->77.90.31.18:4805: use of closed network connection; write tcp 188.166.225.170:52276->77.90.31.18:4805: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52276->77.90.31.18:4805: use of closed network connection Sep 7 10:01:14.850018062 (20h32m50.889591182s) [seg_pos={0 0}, upload=18098]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 7 10:01:14.850083738 (20h32m50.889656858s) [upload=18098]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 7 10:01:14.850087495 (20h32m50.889660615s) [upload=18098]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 7 10:01:15.305723891 (20h32m51.345297021s) [upload=18098]: Done uploading segments... Sep 7 10:01:15.305740914 (20h32m51.345314034s) [upload=18098]: Upload finished. err: <nil> Duration: 6.308321989s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [........X.....X.............................................] duration: 6.35853247s FAIL (2025-09-07 10:34:20 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Sep 7 10:01:12.282591417 (20h32m48.322164577s) [upload=18098]: Starting upload Sep 7 10:01:12.282665129 (20h32m48.322238249s) [upload=18098]: Done starting upload Sep 7 10:01:12.282748708 (20h32m48.322321829s) [upload=18098]: Uploading segments... Sep 7 10:01:12.282962778 (20h32m48.322535898s) [upload=18098]: Got next segment. Inline: false Sep 7 10:01:12.282998316 (20h32m48.322571436s) [upload=18098]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 7 10:01:12.283003185 (20h32m48.322576305s) [upload=18098]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 7 10:01:12.554246002 (20h32m48.593819122s) [seg_pos={0 0}, upload=18098]: Begin upload segment... Sep 7 10:01:12.554288213 (20h32m48.593861343s) [seg_pos={0 0}, upload=18098]: Done begin upload segment. Sep 7 10:01:12.554312018 (20h32m48.593885138s) [noise=true , address=45.91.116.12:4225 , piece_id=24NOULLM6NIKWXYDRUSDNF4ZSNZPDX6GGEATNAGLHFS32STQSFXQ, seg_pos={0 0}, upload=18098]: Uploading piece... Sep 7 10:01:12.554351274 (20h32m48.593924394s) [noise=true , address=45.91.116.8:4045 , piece_id=IYSGEW2LSSMHEGU6PSZWW7Q6DKHQQ4752FWRLHAFWRVDAD266T3Q, seg_pos={0 0}, upload=18098]: Uploading piece... Sep 7 10:01:12.554673811 (20h32m48.594246931s) [noise=true , address=77.90.31.18:4805 , piece_id=XSCLLZAWCOYBF37OZQAWQDGG2BXRDLPFZF7Q45YSPKIB6EDFMTMA, seg_pos={0 0}, upload=18098]: Uploading piece... Sep 7 10:01:12.554765747 (20h32m48.594338867s) [noise=true , address=77.90.31.16:4734 , piece_id=EED3HXQB5DTNMGCF45PYPTSGFDJ7BBE4TLEYVANGIVJKPZBDJXSA, seg_pos={0 0}, upload=18098]: Uploading piece... Sep 7 10:01:12.554935050 (20h32m48.59450817s) [noise=true , address=45.91.116.13:4155 , piece_id=TUEYWZN3DX6LMMINAUNUR2YMKOXC2G3TZ2VSJS27VAT54KJIWOVQ, seg_pos={0 0}, upload=18098]: Uploading piece... Sep 7 10:01:12.555030393 (20h32m48.594603513s) [noise=true , address=77.90.31.10:4520 , piece_id=L45DMYCCMRW3Z6WNFDHA3B347VLU3KNHUOI25TXWKUZXJSNZHAEQ, seg_pos={0 0}, upload=18098]: Uploading piece... Sep 7 10:01:12.555205167 (20h32m48.594778297s) [noise=true , address=45.91.116.4:4939 , piece_id=A7DD37CROE5HDXDGUQ3TY6C6FGQR3M6FY3M4X7GO45I6FBWROS2Q, seg_pos={0 0}, upload=18098]: Uploading piece... Sep 7 10:01:12.555359233 (20h32m48.594932353s) [noise=true , address=77.90.31.15:4699 , piece_id=IGGRTGGDMP66HEJMVPVEKJJPI4KEFC4XUPHBOELDIEJSWRZ4WGBQ, seg_pos={0 0}, upload=18098]: Uploading piece... Sep 7 10:01:12.555428104 (20h32m48.595001224s) [noise=true , address=45.91.116.10:4083 , piece_id=TJGO4QHAPO4FOA5VRFN5KLSW54565U6J3AE4I23WKHG4FRPFR5GA, seg_pos={0 0}, upload=18098]: Uploading piece... Sep 7 10:01:12.555587289 (20h32m48.595160419s) [noise=true , address=77.90.31.19:4841 , piece_id=YCDQ2IPAZILCE6EZE3HEKQNATD33FTQMC6SGJ63ZYOJ3HUC7VFPQ, seg_pos={0 0}, upload=18098]: Uploading piece... Sep 7 10:01:12.555665148 (20h32m48.595238268s) [noise=true , address=45.91.116.7:4011 , piece_id=AVIKSOU2RAJAMTQ3DCQZOQ5ZQQC3Y32SOVYAJMGNHC4QTSMNX55Q, seg_pos={0 0}, upload=18098]: Uploading piece... Sep 7 10:01:12.555961315 (20h32m48.595534435s) [noise=true , address=77.90.31.14:4661 , piece_id=BERKPMDU43TSCVU6W36XW3ZMRSETDIWWZUDWLOUJMD54N5I5R6UA, seg_pos={0 0}, upload=18098]: Uploading piece... Sep 7 10:01:12.555973077 (20h32m48.595546207s) [noise=true , address=77.90.31.12:4592 , piece_id=Y7KQVZZL6VRDVNPXH57UPPVIKNPQBLRCVY7PYUV362QZ6CIGN44A, seg_pos={0 0}, upload=18098]: Uploading piece... Sep 7 10:01:12.556368644 (20h32m48.595941764s) [noise=true , address=77.90.31.12:4590 , piece_id=5VLTXP25Y6JPOHBFKEN57P25J3QEL3F54G6Z7UZQTZDRSQKLC27Q, seg_pos={0 0}, upload=18098]: Uploading piece... Sep 7 10:01:12.556645804 (20h32m48.596218924s) [noise=true , address=77.90.31.19:4842 , piece_id=F6IJY4FN7RI6AIK6AW4E62YPS4A36UMZLMELMQLTPJ7ZZS4UXHCQ, seg_pos={0 0}, upload=18098]: Uploading piece... Sep 7 10:01:12.556871486 (20h32m48.596444606s) [noise=true , address=77.90.31.10:4519 , piece_id=5WDKJ2LFXDDD52BHHPLHFEZKKDSQUINZ5UIICKWXMFZYNY7AFKWA, seg_pos={0 0}, upload=18098]: Uploading piece... Sep 7 10:01:14.384175066 (20h32m50.423748186s) [upload=18098]: Next returned nil segment Sep 7 10:01:14.384226745 (20h32m50.423799875s) [upload=18098]: Waiting for error group managing segments... Sep 7 10:01:14.678022165 (20h32m50.717595295s) [noise=true , address=77.90.31.15:4699 , piece_id=IGGRTGGDMP66HEJMVPVEKJJPI4KEFC4XUPHBOELDIEJSWRZ4WGBQ, seg_pos={0 0}, upload=18098]: Done uploading piece. err: <nil> Sep 7 10:01:14.701635289 (20h32m50.74120841s) [noise=true , address=77.90.31.10:4519 , piece_id=5WDKJ2LFXDDD52BHHPLHFEZKKDSQUINZ5UIICKWXMFZYNY7AFKWA, seg_pos={0 0}, upload=18098]: Done uploading piece. err: <nil> Sep 7 10:01:14.722546303 (20h32m50.762119423s) [noise=true , address=45.91.116.4:4939 , piece_id=A7DD37CROE5HDXDGUQ3TY6C6FGQR3M6FY3M4X7GO45I6FBWROS2Q, seg_pos={0 0}, upload=18098]: Done uploading piece. err: <nil> Sep 7 10:01:14.739108335 (20h32m50.778681455s) [noise=true , address=77.90.31.19:4842 , piece_id=F6IJY4FN7RI6AIK6AW4E62YPS4A36UMZLMELMQLTPJ7ZZS4UXHCQ, seg_pos={0 0}, upload=18098]: Done uploading piece. err: <nil> Sep 7 10:01:14.749560815 (20h32m50.789133935s) [noise=true , address=45.91.116.13:4155 , piece_id=TUEYWZN3DX6LMMINAUNUR2YMKOXC2G3TZ2VSJS27VAT54KJIWOVQ, seg_pos={0 0}, upload=18098]: Done uploading piece. err: <nil> Sep 7 10:01:14.754835469 (20h32m50.794408589s) [noise=true , address=77.90.31.19:4841 , piece_id=YCDQ2IPAZILCE6EZE3HEKQNATD33FTQMC6SGJ63ZYOJ3HUC7VFPQ, seg_pos={0 0}, upload=18098]: Done uploading piece. err: <nil> Sep 7 10:01:14.771610288 (20h32m50.811183418s) [noise=true , address=45.91.116.8:4045 , piece_id=IYSGEW2LSSMHEGU6PSZWW7Q6DKHQQ4752FWRLHAFWRVDAD266T3Q, seg_pos={0 0}, upload=18098]: Done uploading piece. err: <nil> Sep 7 10:01:14.798344604 (20h32m50.837917724s) [noise=true , address=77.90.31.16:4734 , piece_id=EED3HXQB5DTNMGCF45PYPTSGFDJ7BBE4TLEYVANGIVJKPZBDJXSA, seg_pos={0 0}, upload=18098]: Done uploading piece. err: <nil> Sep 7 10:01:14.811486902 (20h32m50.851060022s) [noise=true , address=45.91.116.10:4083 , piece_id=TJGO4QHAPO4FOA5VRFN5KLSW54565U6J3AE4I23WKHG4FRPFR5GA, seg_pos={0 0}, upload=18098]: Done uploading piece. err: <nil> Sep 7 10:01:14.822484987 (20h32m50.862058107s) [noise=true , address=77.90.31.14:4661 , piece_id=BERKPMDU43TSCVU6W36XW3ZMRSETDIWWZUDWLOUJMD54N5I5R6UA, seg_pos={0 0}, upload=18098]: Done uploading piece. err: <nil> Sep 7 10:01:14.824396044 (20h32m50.863969164s) [noise=true , address=77.90.31.12:4590 , piece_id=5VLTXP25Y6JPOHBFKEN57P25J3QEL3F54G6Z7UZQTZDRSQKLC27Q, seg_pos={0 0}, upload=18098]: Done uploading piece. err: <nil> Sep 7 10:01:14.843161772 (20h32m50.882734892s) [noise=true , address=45.91.116.7:4011 , piece_id=AVIKSOU2RAJAMTQ3DCQZOQ5ZQQC3Y32SOVYAJMGNHC4QTSMNX55Q, seg_pos={0 0}, upload=18098]: Done uploading piece. err: <nil> Sep 7 10:01:14.849635850 (20h32m50.88920896s) [noise=true , address=45.91.116.12:4225 , piece_id=24NOULLM6NIKWXYDRUSDNF4ZSNZPDX6GGEATNAGLHFS32STQSFXQ, seg_pos={0 0}, upload=18098]: Done uploading piece. err: <nil> Sep 7 10:01:14.849660828 (20h32m50.889233948s) [seg_pos={0 0}, upload=18098]: Segment reached optimal threshold of 13 pieces. Sep 7 10:01:14.849916888 (20h32m50.889490008s) [noise=true , address=77.90.31.12:4592 , piece_id=Y7KQVZZL6VRDVNPXH57UPPVIKNPQBLRCVY7PYUV362QZ6CIGN44A, seg_pos={0 0}, upload=18098]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3): protocol: expected piece hash; context canceled; EOF Sep 7 10:01:14.849984107 (20h32m50.889557227s) [noise=true , address=77.90.31.10:4520 , piece_id=L45DMYCCMRW3Z6WNFDHA3B347VLU3KNHUOI25TXWKUZXJSNZHAEQ, seg_pos={0 0}, upload=18098]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp): protocol: write tcp 188.166.225.170:42490->77.90.31.10:4520: use of closed network connection; write tcp 188.166.225.170:42490->77.90.31.10:4520: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42490->77.90.31.10:4520: use of closed network connection Sep 7 10:01:14.850004236 (20h32m50.889577356s) [noise=true , address=77.90.31.18:4805 , piece_id=XSCLLZAWCOYBF37OZQAWQDGG2BXRDLPFZF7Q45YSPKIB6EDFMTMA, seg_pos={0 0}, upload=18098]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT): protocol: write tcp 188.166.225.170:52276->77.90.31.18:4805: use of closed network connection; write tcp 188.166.225.170:52276->77.90.31.18:4805: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52276->77.90.31.18:4805: use of closed network connection Sep 7 10:01:14.850018062 (20h32m50.889591182s) [seg_pos={0 0}, upload=18098]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 7 10:01:14.850083738 (20h32m50.889656858s) [upload=18098]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 7 10:01:14.850087495 (20h32m50.889660615s) [upload=18098]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 7 10:01:15.305723891 (20h32m51.345297021s) [upload=18098]: Done uploading segments... Sep 7 10:01:15.305740914 (20h32m51.345314034s) [upload=18098]: Upload finished. err: <nil> Duration: 6.358562769s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.........X.....X............................................] duration: 6.506654053s FAIL (2025-09-07 10:33:48 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Sep 7 10:01:12.282591417 (20h32m48.322164577s) [upload=18098]: Starting upload Sep 7 10:01:12.282665129 (20h32m48.322238249s) [upload=18098]: Done starting upload Sep 7 10:01:12.282748708 (20h32m48.322321829s) [upload=18098]: Uploading segments... Sep 7 10:01:12.282962778 (20h32m48.322535898s) [upload=18098]: Got next segment. Inline: false Sep 7 10:01:12.282998316 (20h32m48.322571436s) [upload=18098]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 7 10:01:12.283003185 (20h32m48.322576305s) [upload=18098]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 7 10:01:12.554246002 (20h32m48.593819122s) [seg_pos={0 0}, upload=18098]: Begin upload segment... Sep 7 10:01:12.554288213 (20h32m48.593861343s) [seg_pos={0 0}, upload=18098]: Done begin upload segment. Sep 7 10:01:12.554312018 (20h32m48.593885138s) [noise=true , address=45.91.116.12:4225 , piece_id=24NOULLM6NIKWXYDRUSDNF4ZSNZPDX6GGEATNAGLHFS32STQSFXQ, seg_pos={0 0}, upload=18098]: Uploading piece... Sep 7 10:01:12.554351274 (20h32m48.593924394s) [noise=true , address=45.91.116.8:4045 , piece_id=IYSGEW2LSSMHEGU6PSZWW7Q6DKHQQ4752FWRLHAFWRVDAD266T3Q, seg_pos={0 0}, upload=18098]: Uploading piece... Sep 7 10:01:12.554673811 (20h32m48.594246931s) [noise=true , address=77.90.31.18:4805 , piece_id=XSCLLZAWCOYBF37OZQAWQDGG2BXRDLPFZF7Q45YSPKIB6EDFMTMA, seg_pos={0 0}, upload=18098]: Uploading piece... Sep 7 10:01:12.554765747 (20h32m48.594338867s) [noise=true , address=77.90.31.16:4734 , piece_id=EED3HXQB5DTNMGCF45PYPTSGFDJ7BBE4TLEYVANGIVJKPZBDJXSA, seg_pos={0 0}, upload=18098]: Uploading piece... Sep 7 10:01:12.554935050 (20h32m48.59450817s) [noise=true , address=45.91.116.13:4155 , piece_id=TUEYWZN3DX6LMMINAUNUR2YMKOXC2G3TZ2VSJS27VAT54KJIWOVQ, seg_pos={0 0}, upload=18098]: Uploading piece... Sep 7 10:01:12.555030393 (20h32m48.594603513s) [noise=true , address=77.90.31.10:4520 , piece_id=L45DMYCCMRW3Z6WNFDHA3B347VLU3KNHUOI25TXWKUZXJSNZHAEQ, seg_pos={0 0}, upload=18098]: Uploading piece... Sep 7 10:01:12.555205167 (20h32m48.594778297s) [noise=true , address=45.91.116.4:4939 , piece_id=A7DD37CROE5HDXDGUQ3TY6C6FGQR3M6FY3M4X7GO45I6FBWROS2Q, seg_pos={0 0}, upload=18098]: Uploading piece... Sep 7 10:01:12.555359233 (20h32m48.594932353s) [noise=true , address=77.90.31.15:4699 , piece_id=IGGRTGGDMP66HEJMVPVEKJJPI4KEFC4XUPHBOELDIEJSWRZ4WGBQ, seg_pos={0 0}, upload=18098]: Uploading piece... Sep 7 10:01:12.555428104 (20h32m48.595001224s) [noise=true , address=45.91.116.10:4083 , piece_id=TJGO4QHAPO4FOA5VRFN5KLSW54565U6J3AE4I23WKHG4FRPFR5GA, seg_pos={0 0}, upload=18098]: Uploading piece... Sep 7 10:01:12.555587289 (20h32m48.595160419s) [noise=true , address=77.90.31.19:4841 , piece_id=YCDQ2IPAZILCE6EZE3HEKQNATD33FTQMC6SGJ63ZYOJ3HUC7VFPQ, seg_pos={0 0}, upload=18098]: Uploading piece... Sep 7 10:01:12.555665148 (20h32m48.595238268s) [noise=true , address=45.91.116.7:4011 , piece_id=AVIKSOU2RAJAMTQ3DCQZOQ5ZQQC3Y32SOVYAJMGNHC4QTSMNX55Q, seg_pos={0 0}, upload=18098]: Uploading piece... Sep 7 10:01:12.555961315 (20h32m48.595534435s) [noise=true , address=77.90.31.14:4661 , piece_id=BERKPMDU43TSCVU6W36XW3ZMRSETDIWWZUDWLOUJMD54N5I5R6UA, seg_pos={0 0}, upload=18098]: Uploading piece... Sep 7 10:01:12.555973077 (20h32m48.595546207s) [noise=true , address=77.90.31.12:4592 , piece_id=Y7KQVZZL6VRDVNPXH57UPPVIKNPQBLRCVY7PYUV362QZ6CIGN44A, seg_pos={0 0}, upload=18098]: Uploading piece... Sep 7 10:01:12.556368644 (20h32m48.595941764s) [noise=true , address=77.90.31.12:4590 , piece_id=5VLTXP25Y6JPOHBFKEN57P25J3QEL3F54G6Z7UZQTZDRSQKLC27Q, seg_pos={0 0}, upload=18098]: Uploading piece... Sep 7 10:01:12.556645804 (20h32m48.596218924s) [noise=true , address=77.90.31.19:4842 , piece_id=F6IJY4FN7RI6AIK6AW4E62YPS4A36UMZLMELMQLTPJ7ZZS4UXHCQ, seg_pos={0 0}, upload=18098]: Uploading piece... Sep 7 10:01:12.556871486 (20h32m48.596444606s) [noise=true , address=77.90.31.10:4519 , piece_id=5WDKJ2LFXDDD52BHHPLHFEZKKDSQUINZ5UIICKWXMFZYNY7AFKWA, seg_pos={0 0}, upload=18098]: Uploading piece... Sep 7 10:01:14.384175066 (20h32m50.423748186s) [upload=18098]: Next returned nil segment Sep 7 10:01:14.384226745 (20h32m50.423799875s) [upload=18098]: Waiting for error group managing segments... Sep 7 10:01:14.678022165 (20h32m50.717595295s) [noise=true , address=77.90.31.15:4699 , piece_id=IGGRTGGDMP66HEJMVPVEKJJPI4KEFC4XUPHBOELDIEJSWRZ4WGBQ, seg_pos={0 0}, upload=18098]: Done uploading piece. err: <nil> Sep 7 10:01:14.701635289 (20h32m50.74120841s) [noise=true , address=77.90.31.10:4519 , piece_id=5WDKJ2LFXDDD52BHHPLHFEZKKDSQUINZ5UIICKWXMFZYNY7AFKWA, seg_pos={0 0}, upload=18098]: Done uploading piece. err: <nil> Sep 7 10:01:14.722546303 (20h32m50.762119423s) [noise=true , address=45.91.116.4:4939 , piece_id=A7DD37CROE5HDXDGUQ3TY6C6FGQR3M6FY3M4X7GO45I6FBWROS2Q, seg_pos={0 0}, upload=18098]: Done uploading piece. err: <nil> Sep 7 10:01:14.739108335 (20h32m50.778681455s) [noise=true , address=77.90.31.19:4842 , piece_id=F6IJY4FN7RI6AIK6AW4E62YPS4A36UMZLMELMQLTPJ7ZZS4UXHCQ, seg_pos={0 0}, upload=18098]: Done uploading piece. err: <nil> Sep 7 10:01:14.749560815 (20h32m50.789133935s) [noise=true , address=45.91.116.13:4155 , piece_id=TUEYWZN3DX6LMMINAUNUR2YMKOXC2G3TZ2VSJS27VAT54KJIWOVQ, seg_pos={0 0}, upload=18098]: Done uploading piece. err: <nil> Sep 7 10:01:14.754835469 (20h32m50.794408589s) [noise=true , address=77.90.31.19:4841 , piece_id=YCDQ2IPAZILCE6EZE3HEKQNATD33FTQMC6SGJ63ZYOJ3HUC7VFPQ, seg_pos={0 0}, upload=18098]: Done uploading piece. err: <nil> Sep 7 10:01:14.771610288 (20h32m50.811183418s) [noise=true , address=45.91.116.8:4045 , piece_id=IYSGEW2LSSMHEGU6PSZWW7Q6DKHQQ4752FWRLHAFWRVDAD266T3Q, seg_pos={0 0}, upload=18098]: Done uploading piece. err: <nil> Sep 7 10:01:14.798344604 (20h32m50.837917724s) [noise=true , address=77.90.31.16:4734 , piece_id=EED3HXQB5DTNMGCF45PYPTSGFDJ7BBE4TLEYVANGIVJKPZBDJXSA, seg_pos={0 0}, upload=18098]: Done uploading piece. err: <nil> Sep 7 10:01:14.811486902 (20h32m50.851060022s) [noise=true , address=45.91.116.10:4083 , piece_id=TJGO4QHAPO4FOA5VRFN5KLSW54565U6J3AE4I23WKHG4FRPFR5GA, seg_pos={0 0}, upload=18098]: Done uploading piece. err: <nil> Sep 7 10:01:14.822484987 (20h32m50.862058107s) [noise=true , address=77.90.31.14:4661 , piece_id=BERKPMDU43TSCVU6W36XW3ZMRSETDIWWZUDWLOUJMD54N5I5R6UA, seg_pos={0 0}, upload=18098]: Done uploading piece. err: <nil> Sep 7 10:01:14.824396044 (20h32m50.863969164s) [noise=true , address=77.90.31.12:4590 , piece_id=5VLTXP25Y6JPOHBFKEN57P25J3QEL3F54G6Z7UZQTZDRSQKLC27Q, seg_pos={0 0}, upload=18098]: Done uploading piece. err: <nil> Sep 7 10:01:14.843161772 (20h32m50.882734892s) [noise=true , address=45.91.116.7:4011 , piece_id=AVIKSOU2RAJAMTQ3DCQZOQ5ZQQC3Y32SOVYAJMGNHC4QTSMNX55Q, seg_pos={0 0}, upload=18098]: Done uploading piece. err: <nil> Sep 7 10:01:14.849635850 (20h32m50.88920896s) [noise=true , address=45.91.116.12:4225 , piece_id=24NOULLM6NIKWXYDRUSDNF4ZSNZPDX6GGEATNAGLHFS32STQSFXQ, seg_pos={0 0}, upload=18098]: Done uploading piece. err: <nil> Sep 7 10:01:14.849660828 (20h32m50.889233948s) [seg_pos={0 0}, upload=18098]: Segment reached optimal threshold of 13 pieces. Sep 7 10:01:14.849916888 (20h32m50.889490008s) [noise=true , address=77.90.31.12:4592 , piece_id=Y7KQVZZL6VRDVNPXH57UPPVIKNPQBLRCVY7PYUV362QZ6CIGN44A, seg_pos={0 0}, upload=18098]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3): protocol: expected piece hash; context canceled; EOF Sep 7 10:01:14.849984107 (20h32m50.889557227s) [noise=true , address=77.90.31.10:4520 , piece_id=L45DMYCCMRW3Z6WNFDHA3B347VLU3KNHUOI25TXWKUZXJSNZHAEQ, seg_pos={0 0}, upload=18098]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp): protocol: write tcp 188.166.225.170:42490->77.90.31.10:4520: use of closed network connection; write tcp 188.166.225.170:42490->77.90.31.10:4520: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42490->77.90.31.10:4520: use of closed network connection Sep 7 10:01:14.850004236 (20h32m50.889577356s) [noise=true , address=77.90.31.18:4805 , piece_id=XSCLLZAWCOYBF37OZQAWQDGG2BXRDLPFZF7Q45YSPKIB6EDFMTMA, seg_pos={0 0}, upload=18098]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT): protocol: write tcp 188.166.225.170:52276->77.90.31.18:4805: use of closed network connection; write tcp 188.166.225.170:52276->77.90.31.18:4805: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52276->77.90.31.18:4805: use of closed network connection Sep 7 10:01:14.850018062 (20h32m50.889591182s) [seg_pos={0 0}, upload=18098]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 7 10:01:14.850083738 (20h32m50.889656858s) [upload=18098]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 7 10:01:14.850087495 (20h32m50.889660615s) [upload=18098]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 7 10:01:15.305723891 (20h32m51.345297021s) [upload=18098]: Done uploading segments... Sep 7 10:01:15.305740914 (20h32m51.345314034s) [upload=18098]: Upload finished. err: <nil> Duration: 6.506674001s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..........X.....X...........................................] duration: 8.891289762s FAIL (2025-09-07 10:33:19 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Sep 7 10:01:12.282591417 (20h32m48.322164577s) [upload=18098]: Starting upload Sep 7 10:01:12.282665129 (20h32m48.322238249s) [upload=18098]: Done starting upload Sep 7 10:01:12.282748708 (20h32m48.322321829s) [upload=18098]: Uploading segments... Sep 7 10:01:12.282962778 (20h32m48.322535898s) [upload=18098]: Got next segment. Inline: false Sep 7 10:01:12.282998316 (20h32m48.322571436s) [upload=18098]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 7 10:01:12.283003185 (20h32m48.322576305s) [upload=18098]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 7 10:01:12.554246002 (20h32m48.593819122s) [seg_pos={0 0}, upload=18098]: Begin upload segment... Sep 7 10:01:12.554288213 (20h32m48.593861343s) [seg_pos={0 0}, upload=18098]: Done begin upload segment. Sep 7 10:01:12.554312018 (20h32m48.593885138s) [noise=true , address=45.91.116.12:4225 , piece_id=24NOULLM6NIKWXYDRUSDNF4ZSNZPDX6GGEATNAGLHFS32STQSFXQ, seg_pos={0 0}, upload=18098]: Uploading piece... Sep 7 10:01:12.554351274 (20h32m48.593924394s) [noise=true , address=45.91.116.8:4045 , piece_id=IYSGEW2LSSMHEGU6PSZWW7Q6DKHQQ4752FWRLHAFWRVDAD266T3Q, seg_pos={0 0}, upload=18098]: Uploading piece... Sep 7 10:01:12.554673811 (20h32m48.594246931s) [noise=true , address=77.90.31.18:4805 , piece_id=XSCLLZAWCOYBF37OZQAWQDGG2BXRDLPFZF7Q45YSPKIB6EDFMTMA, seg_pos={0 0}, upload=18098]: Uploading piece... Sep 7 10:01:12.554765747 (20h32m48.594338867s) [noise=true , address=77.90.31.16:4734 , piece_id=EED3HXQB5DTNMGCF45PYPTSGFDJ7BBE4TLEYVANGIVJKPZBDJXSA, seg_pos={0 0}, upload=18098]: Uploading piece... Sep 7 10:01:12.554935050 (20h32m48.59450817s) [noise=true , address=45.91.116.13:4155 , piece_id=TUEYWZN3DX6LMMINAUNUR2YMKOXC2G3TZ2VSJS27VAT54KJIWOVQ, seg_pos={0 0}, upload=18098]: Uploading piece... Sep 7 10:01:12.555030393 (20h32m48.594603513s) [noise=true , address=77.90.31.10:4520 , piece_id=L45DMYCCMRW3Z6WNFDHA3B347VLU3KNHUOI25TXWKUZXJSNZHAEQ, seg_pos={0 0}, upload=18098]: Uploading piece... Sep 7 10:01:12.555205167 (20h32m48.594778297s) [noise=true , address=45.91.116.4:4939 , piece_id=A7DD37CROE5HDXDGUQ3TY6C6FGQR3M6FY3M4X7GO45I6FBWROS2Q, seg_pos={0 0}, upload=18098]: Uploading piece... Sep 7 10:01:12.555359233 (20h32m48.594932353s) [noise=true , address=77.90.31.15:4699 , piece_id=IGGRTGGDMP66HEJMVPVEKJJPI4KEFC4XUPHBOELDIEJSWRZ4WGBQ, seg_pos={0 0}, upload=18098]: Uploading piece... Sep 7 10:01:12.555428104 (20h32m48.595001224s) [noise=true , address=45.91.116.10:4083 , piece_id=TJGO4QHAPO4FOA5VRFN5KLSW54565U6J3AE4I23WKHG4FRPFR5GA, seg_pos={0 0}, upload=18098]: Uploading piece... Sep 7 10:01:12.555587289 (20h32m48.595160419s) [noise=true , address=77.90.31.19:4841 , piece_id=YCDQ2IPAZILCE6EZE3HEKQNATD33FTQMC6SGJ63ZYOJ3HUC7VFPQ, seg_pos={0 0}, upload=18098]: Uploading piece... Sep 7 10:01:12.555665148 (20h32m48.595238268s) [noise=true , address=45.91.116.7:4011 , piece_id=AVIKSOU2RAJAMTQ3DCQZOQ5ZQQC3Y32SOVYAJMGNHC4QTSMNX55Q, seg_pos={0 0}, upload=18098]: Uploading piece... Sep 7 10:01:12.555961315 (20h32m48.595534435s) [noise=true , address=77.90.31.14:4661 , piece_id=BERKPMDU43TSCVU6W36XW3ZMRSETDIWWZUDWLOUJMD54N5I5R6UA, seg_pos={0 0}, upload=18098]: Uploading piece... Sep 7 10:01:12.555973077 (20h32m48.595546207s) [noise=true , address=77.90.31.12:4592 , piece_id=Y7KQVZZL6VRDVNPXH57UPPVIKNPQBLRCVY7PYUV362QZ6CIGN44A, seg_pos={0 0}, upload=18098]: Uploading piece... Sep 7 10:01:12.556368644 (20h32m48.595941764s) [noise=true , address=77.90.31.12:4590 , piece_id=5VLTXP25Y6JPOHBFKEN57P25J3QEL3F54G6Z7UZQTZDRSQKLC27Q, seg_pos={0 0}, upload=18098]: Uploading piece... Sep 7 10:01:12.556645804 (20h32m48.596218924s) [noise=true , address=77.90.31.19:4842 , piece_id=F6IJY4FN7RI6AIK6AW4E62YPS4A36UMZLMELMQLTPJ7ZZS4UXHCQ, seg_pos={0 0}, upload=18098]: Uploading piece... Sep 7 10:01:12.556871486 (20h32m48.596444606s) [noise=true , address=77.90.31.10:4519 , piece_id=5WDKJ2LFXDDD52BHHPLHFEZKKDSQUINZ5UIICKWXMFZYNY7AFKWA, seg_pos={0 0}, upload=18098]: Uploading piece... Sep 7 10:01:14.384175066 (20h32m50.423748186s) [upload=18098]: Next returned nil segment Sep 7 10:01:14.384226745 (20h32m50.423799875s) [upload=18098]: Waiting for error group managing segments... Sep 7 10:01:14.678022165 (20h32m50.717595295s) [noise=true , address=77.90.31.15:4699 , piece_id=IGGRTGGDMP66HEJMVPVEKJJPI4KEFC4XUPHBOELDIEJSWRZ4WGBQ, seg_pos={0 0}, upload=18098]: Done uploading piece. err: <nil> Sep 7 10:01:14.701635289 (20h32m50.74120841s) [noise=true , address=77.90.31.10:4519 , piece_id=5WDKJ2LFXDDD52BHHPLHFEZKKDSQUINZ5UIICKWXMFZYNY7AFKWA, seg_pos={0 0}, upload=18098]: Done uploading piece. err: <nil> Sep 7 10:01:14.722546303 (20h32m50.762119423s) [noise=true , address=45.91.116.4:4939 , piece_id=A7DD37CROE5HDXDGUQ3TY6C6FGQR3M6FY3M4X7GO45I6FBWROS2Q, seg_pos={0 0}, upload=18098]: Done uploading piece. err: <nil> Sep 7 10:01:14.739108335 (20h32m50.778681455s) [noise=true , address=77.90.31.19:4842 , piece_id=F6IJY4FN7RI6AIK6AW4E62YPS4A36UMZLMELMQLTPJ7ZZS4UXHCQ, seg_pos={0 0}, upload=18098]: Done uploading piece. err: <nil> Sep 7 10:01:14.749560815 (20h32m50.789133935s) [noise=true , address=45.91.116.13:4155 , piece_id=TUEYWZN3DX6LMMINAUNUR2YMKOXC2G3TZ2VSJS27VAT54KJIWOVQ, seg_pos={0 0}, upload=18098]: Done uploading piece. err: <nil> Sep 7 10:01:14.754835469 (20h32m50.794408589s) [noise=true , address=77.90.31.19:4841 , piece_id=YCDQ2IPAZILCE6EZE3HEKQNATD33FTQMC6SGJ63ZYOJ3HUC7VFPQ, seg_pos={0 0}, upload=18098]: Done uploading piece. err: <nil> Sep 7 10:01:14.771610288 (20h32m50.811183418s) [noise=true , address=45.91.116.8:4045 , piece_id=IYSGEW2LSSMHEGU6PSZWW7Q6DKHQQ4752FWRLHAFWRVDAD266T3Q, seg_pos={0 0}, upload=18098]: Done uploading piece. err: <nil> Sep 7 10:01:14.798344604 (20h32m50.837917724s) [noise=true , address=77.90.31.16:4734 , piece_id=EED3HXQB5DTNMGCF45PYPTSGFDJ7BBE4TLEYVANGIVJKPZBDJXSA, seg_pos={0 0}, upload=18098]: Done uploading piece. err: <nil> Sep 7 10:01:14.811486902 (20h32m50.851060022s) [noise=true , address=45.91.116.10:4083 , piece_id=TJGO4QHAPO4FOA5VRFN5KLSW54565U6J3AE4I23WKHG4FRPFR5GA, seg_pos={0 0}, upload=18098]: Done uploading piece. err: <nil> Sep 7 10:01:14.822484987 (20h32m50.862058107s) [noise=true , address=77.90.31.14:4661 , piece_id=BERKPMDU43TSCVU6W36XW3ZMRSETDIWWZUDWLOUJMD54N5I5R6UA, seg_pos={0 0}, upload=18098]: Done uploading piece. err: <nil> Sep 7 10:01:14.824396044 (20h32m50.863969164s) [noise=true , address=77.90.31.12:4590 , piece_id=5VLTXP25Y6JPOHBFKEN57P25J3QEL3F54G6Z7UZQTZDRSQKLC27Q, seg_pos={0 0}, upload=18098]: Done uploading piece. err: <nil> Sep 7 10:01:14.843161772 (20h32m50.882734892s) [noise=true , address=45.91.116.7:4011 , piece_id=AVIKSOU2RAJAMTQ3DCQZOQ5ZQQC3Y32SOVYAJMGNHC4QTSMNX55Q, seg_pos={0 0}, upload=18098]: Done uploading piece. err: <nil> Sep 7 10:01:14.849635850 (20h32m50.88920896s) [noise=true , address=45.91.116.12:4225 , piece_id=24NOULLM6NIKWXYDRUSDNF4ZSNZPDX6GGEATNAGLHFS32STQSFXQ, seg_pos={0 0}, upload=18098]: Done uploading piece. err: <nil> Sep 7 10:01:14.849660828 (20h32m50.889233948s) [seg_pos={0 0}, upload=18098]: Segment reached optimal threshold of 13 pieces. Sep 7 10:01:14.849916888 (20h32m50.889490008s) [noise=true , address=77.90.31.12:4592 , piece_id=Y7KQVZZL6VRDVNPXH57UPPVIKNPQBLRCVY7PYUV362QZ6CIGN44A, seg_pos={0 0}, upload=18098]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3): protocol: expected piece hash; context canceled; EOF Sep 7 10:01:14.849984107 (20h32m50.889557227s) [noise=true , address=77.90.31.10:4520 , piece_id=L45DMYCCMRW3Z6WNFDHA3B347VLU3KNHUOI25TXWKUZXJSNZHAEQ, seg_pos={0 0}, upload=18098]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp): protocol: write tcp 188.166.225.170:42490->77.90.31.10:4520: use of closed network connection; write tcp 188.166.225.170:42490->77.90.31.10:4520: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42490->77.90.31.10:4520: use of closed network connection Sep 7 10:01:14.850004236 (20h32m50.889577356s) [noise=true , address=77.90.31.18:4805 , piece_id=XSCLLZAWCOYBF37OZQAWQDGG2BXRDLPFZF7Q45YSPKIB6EDFMTMA, seg_pos={0 0}, upload=18098]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT): protocol: write tcp 188.166.225.170:52276->77.90.31.18:4805: use of closed network connection; write tcp 188.166.225.170:52276->77.90.31.18:4805: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52276->77.90.31.18:4805: use of closed network connection Sep 7 10:01:14.850018062 (20h32m50.889591182s) [seg_pos={0 0}, upload=18098]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 7 10:01:14.850083738 (20h32m50.889656858s) [upload=18098]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 7 10:01:14.850087495 (20h32m50.889660615s) [upload=18098]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 7 10:01:15.305723891 (20h32m51.345297021s) [upload=18098]: Done uploading segments... Sep 7 10:01:15.305740914 (20h32m51.345314034s) [upload=18098]: Upload finished. err: <nil> Duration: 8.891321723s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...........X.....X..........................................] duration: 6.394367921s FAIL (2025-09-07 10:32:31 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Sep 7 10:01:12.282591417 (20h32m48.322164577s) [upload=18098]: Starting upload Sep 7 10:01:12.282665129 (20h32m48.322238249s) [upload=18098]: Done starting upload Sep 7 10:01:12.282748708 (20h32m48.322321829s) [upload=18098]: Uploading segments... Sep 7 10:01:12.282962778 (20h32m48.322535898s) [upload=18098]: Got next segment. Inline: false Sep 7 10:01:12.282998316 (20h32m48.322571436s) [upload=18098]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 7 10:01:12.283003185 (20h32m48.322576305s) [upload=18098]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 7 10:01:12.554246002 (20h32m48.593819122s) [seg_pos={0 0}, upload=18098]: Begin upload segment... Sep 7 10:01:12.554288213 (20h32m48.593861343s) [seg_pos={0 0}, upload=18098]: Done begin upload segment. Sep 7 10:01:12.554312018 (20h32m48.593885138s) [noise=true , address=45.91.116.12:4225 , piece_id=24NOULLM6NIKWXYDRUSDNF4ZSNZPDX6GGEATNAGLHFS32STQSFXQ, seg_pos={0 0}, upload=18098]: Uploading piece... Sep 7 10:01:12.554351274 (20h32m48.593924394s) [noise=true , address=45.91.116.8:4045 , piece_id=IYSGEW2LSSMHEGU6PSZWW7Q6DKHQQ4752FWRLHAFWRVDAD266T3Q, seg_pos={0 0}, upload=18098]: Uploading piece... Sep 7 10:01:12.554673811 (20h32m48.594246931s) [noise=true , address=77.90.31.18:4805 , piece_id=XSCLLZAWCOYBF37OZQAWQDGG2BXRDLPFZF7Q45YSPKIB6EDFMTMA, seg_pos={0 0}, upload=18098]: Uploading piece... Sep 7 10:01:12.554765747 (20h32m48.594338867s) [noise=true , address=77.90.31.16:4734 , piece_id=EED3HXQB5DTNMGCF45PYPTSGFDJ7BBE4TLEYVANGIVJKPZBDJXSA, seg_pos={0 0}, upload=18098]: Uploading piece... Sep 7 10:01:12.554935050 (20h32m48.59450817s) [noise=true , address=45.91.116.13:4155 , piece_id=TUEYWZN3DX6LMMINAUNUR2YMKOXC2G3TZ2VSJS27VAT54KJIWOVQ, seg_pos={0 0}, upload=18098]: Uploading piece... Sep 7 10:01:12.555030393 (20h32m48.594603513s) [noise=true , address=77.90.31.10:4520 , piece_id=L45DMYCCMRW3Z6WNFDHA3B347VLU3KNHUOI25TXWKUZXJSNZHAEQ, seg_pos={0 0}, upload=18098]: Uploading piece... Sep 7 10:01:12.555205167 (20h32m48.594778297s) [noise=true , address=45.91.116.4:4939 , piece_id=A7DD37CROE5HDXDGUQ3TY6C6FGQR3M6FY3M4X7GO45I6FBWROS2Q, seg_pos={0 0}, upload=18098]: Uploading piece... Sep 7 10:01:12.555359233 (20h32m48.594932353s) [noise=true , address=77.90.31.15:4699 , piece_id=IGGRTGGDMP66HEJMVPVEKJJPI4KEFC4XUPHBOELDIEJSWRZ4WGBQ, seg_pos={0 0}, upload=18098]: Uploading piece... Sep 7 10:01:12.555428104 (20h32m48.595001224s) [noise=true , address=45.91.116.10:4083 , piece_id=TJGO4QHAPO4FOA5VRFN5KLSW54565U6J3AE4I23WKHG4FRPFR5GA, seg_pos={0 0}, upload=18098]: Uploading piece... Sep 7 10:01:12.555587289 (20h32m48.595160419s) [noise=true , address=77.90.31.19:4841 , piece_id=YCDQ2IPAZILCE6EZE3HEKQNATD33FTQMC6SGJ63ZYOJ3HUC7VFPQ, seg_pos={0 0}, upload=18098]: Uploading piece... Sep 7 10:01:12.555665148 (20h32m48.595238268s) [noise=true , address=45.91.116.7:4011 , piece_id=AVIKSOU2RAJAMTQ3DCQZOQ5ZQQC3Y32SOVYAJMGNHC4QTSMNX55Q, seg_pos={0 0}, upload=18098]: Uploading piece... Sep 7 10:01:12.555961315 (20h32m48.595534435s) [noise=true , address=77.90.31.14:4661 , piece_id=BERKPMDU43TSCVU6W36XW3ZMRSETDIWWZUDWLOUJMD54N5I5R6UA, seg_pos={0 0}, upload=18098]: Uploading piece... Sep 7 10:01:12.555973077 (20h32m48.595546207s) [noise=true , address=77.90.31.12:4592 , piece_id=Y7KQVZZL6VRDVNPXH57UPPVIKNPQBLRCVY7PYUV362QZ6CIGN44A, seg_pos={0 0}, upload=18098]: Uploading piece... Sep 7 10:01:12.556368644 (20h32m48.595941764s) [noise=true , address=77.90.31.12:4590 , piece_id=5VLTXP25Y6JPOHBFKEN57P25J3QEL3F54G6Z7UZQTZDRSQKLC27Q, seg_pos={0 0}, upload=18098]: Uploading piece... Sep 7 10:01:12.556645804 (20h32m48.596218924s) [noise=true , address=77.90.31.19:4842 , piece_id=F6IJY4FN7RI6AIK6AW4E62YPS4A36UMZLMELMQLTPJ7ZZS4UXHCQ, seg_pos={0 0}, upload=18098]: Uploading piece... Sep 7 10:01:12.556871486 (20h32m48.596444606s) [noise=true , address=77.90.31.10:4519 , piece_id=5WDKJ2LFXDDD52BHHPLHFEZKKDSQUINZ5UIICKWXMFZYNY7AFKWA, seg_pos={0 0}, upload=18098]: Uploading piece... Sep 7 10:01:14.384175066 (20h32m50.423748186s) [upload=18098]: Next returned nil segment Sep 7 10:01:14.384226745 (20h32m50.423799875s) [upload=18098]: Waiting for error group managing segments... Sep 7 10:01:14.678022165 (20h32m50.717595295s) [noise=true , address=77.90.31.15:4699 , piece_id=IGGRTGGDMP66HEJMVPVEKJJPI4KEFC4XUPHBOELDIEJSWRZ4WGBQ, seg_pos={0 0}, upload=18098]: Done uploading piece. err: <nil> Sep 7 10:01:14.701635289 (20h32m50.74120841s) [noise=true , address=77.90.31.10:4519 , piece_id=5WDKJ2LFXDDD52BHHPLHFEZKKDSQUINZ5UIICKWXMFZYNY7AFKWA, seg_pos={0 0}, upload=18098]: Done uploading piece. err: <nil> Sep 7 10:01:14.722546303 (20h32m50.762119423s) [noise=true , address=45.91.116.4:4939 , piece_id=A7DD37CROE5HDXDGUQ3TY6C6FGQR3M6FY3M4X7GO45I6FBWROS2Q, seg_pos={0 0}, upload=18098]: Done uploading piece. err: <nil> Sep 7 10:01:14.739108335 (20h32m50.778681455s) [noise=true , address=77.90.31.19:4842 , piece_id=F6IJY4FN7RI6AIK6AW4E62YPS4A36UMZLMELMQLTPJ7ZZS4UXHCQ, seg_pos={0 0}, upload=18098]: Done uploading piece. err: <nil> Sep 7 10:01:14.749560815 (20h32m50.789133935s) [noise=true , address=45.91.116.13:4155 , piece_id=TUEYWZN3DX6LMMINAUNUR2YMKOXC2G3TZ2VSJS27VAT54KJIWOVQ, seg_pos={0 0}, upload=18098]: Done uploading piece. err: <nil> Sep 7 10:01:14.754835469 (20h32m50.794408589s) [noise=true , address=77.90.31.19:4841 , piece_id=YCDQ2IPAZILCE6EZE3HEKQNATD33FTQMC6SGJ63ZYOJ3HUC7VFPQ, seg_pos={0 0}, upload=18098]: Done uploading piece. err: <nil> Sep 7 10:01:14.771610288 (20h32m50.811183418s) [noise=true , address=45.91.116.8:4045 , piece_id=IYSGEW2LSSMHEGU6PSZWW7Q6DKHQQ4752FWRLHAFWRVDAD266T3Q, seg_pos={0 0}, upload=18098]: Done uploading piece. err: <nil> Sep 7 10:01:14.798344604 (20h32m50.837917724s) [noise=true , address=77.90.31.16:4734 , piece_id=EED3HXQB5DTNMGCF45PYPTSGFDJ7BBE4TLEYVANGIVJKPZBDJXSA, seg_pos={0 0}, upload=18098]: Done uploading piece. err: <nil> Sep 7 10:01:14.811486902 (20h32m50.851060022s) [noise=true , address=45.91.116.10:4083 , piece_id=TJGO4QHAPO4FOA5VRFN5KLSW54565U6J3AE4I23WKHG4FRPFR5GA, seg_pos={0 0}, upload=18098]: Done uploading piece. err: <nil> Sep 7 10:01:14.822484987 (20h32m50.862058107s) [noise=true , address=77.90.31.14:4661 , piece_id=BERKPMDU43TSCVU6W36XW3ZMRSETDIWWZUDWLOUJMD54N5I5R6UA, seg_pos={0 0}, upload=18098]: Done uploading piece. err: <nil> Sep 7 10:01:14.824396044 (20h32m50.863969164s) [noise=true , address=77.90.31.12:4590 , piece_id=5VLTXP25Y6JPOHBFKEN57P25J3QEL3F54G6Z7UZQTZDRSQKLC27Q, seg_pos={0 0}, upload=18098]: Done uploading piece. err: <nil> Sep 7 10:01:14.843161772 (20h32m50.882734892s) [noise=true , address=45.91.116.7:4011 , piece_id=AVIKSOU2RAJAMTQ3DCQZOQ5ZQQC3Y32SOVYAJMGNHC4QTSMNX55Q, seg_pos={0 0}, upload=18098]: Done uploading piece. err: <nil> Sep 7 10:01:14.849635850 (20h32m50.88920896s) [noise=true , address=45.91.116.12:4225 , piece_id=24NOULLM6NIKWXYDRUSDNF4ZSNZPDX6GGEATNAGLHFS32STQSFXQ, seg_pos={0 0}, upload=18098]: Done uploading piece. err: <nil> Sep 7 10:01:14.849660828 (20h32m50.889233948s) [seg_pos={0 0}, upload=18098]: Segment reached optimal threshold of 13 pieces. Sep 7 10:01:14.849916888 (20h32m50.889490008s) [noise=true , address=77.90.31.12:4592 , piece_id=Y7KQVZZL6VRDVNPXH57UPPVIKNPQBLRCVY7PYUV362QZ6CIGN44A, seg_pos={0 0}, upload=18098]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3): protocol: expected piece hash; context canceled; EOF Sep 7 10:01:14.849984107 (20h32m50.889557227s) [noise=true , address=77.90.31.10:4520 , piece_id=L45DMYCCMRW3Z6WNFDHA3B347VLU3KNHUOI25TXWKUZXJSNZHAEQ, seg_pos={0 0}, upload=18098]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp): protocol: write tcp 188.166.225.170:42490->77.90.31.10:4520: use of closed network connection; write tcp 188.166.225.170:42490->77.90.31.10:4520: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42490->77.90.31.10:4520: use of closed network connection Sep 7 10:01:14.850004236 (20h32m50.889577356s) [noise=true , address=77.90.31.18:4805 , piece_id=XSCLLZAWCOYBF37OZQAWQDGG2BXRDLPFZF7Q45YSPKIB6EDFMTMA, seg_pos={0 0}, upload=18098]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT): protocol: write tcp 188.166.225.170:52276->77.90.31.18:4805: use of closed network connection; write tcp 188.166.225.170:52276->77.90.31.18:4805: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52276->77.90.31.18:4805: use of closed network connection Sep 7 10:01:14.850018062 (20h32m50.889591182s) [seg_pos={0 0}, upload=18098]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 7 10:01:14.850083738 (20h32m50.889656858s) [upload=18098]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 7 10:01:14.850087495 (20h32m50.889660615s) [upload=18098]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 7 10:01:15.305723891 (20h32m51.345297021s) [upload=18098]: Done uploading segments... Sep 7 10:01:15.305740914 (20h32m51.345314034s) [upload=18098]: Upload finished. err: <nil> Duration: 6.394408069s core uplink us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)