Orbiter History

checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 13 out of 60 is failed (threshold=6) [X.XX.......X..XX......X.X...X.......X...........X.X....X....] duration: 3.447199388s trace: https://jaeger.dev.storj.io/trace/102f7d28bf9f3520 FAIL (2025-11-22 16:42:35 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.447228282s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............X............................................] duration: 12.623148362s trace: https://jaeger.dev.storj.io/trace/13bcd3d57614c8f8 FAIL (2025-11-22 16:42:25 UTC) orbiter warning: context deadline exceeded Duration: 12.623220158s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.....................XX....................................X] duration: 30.486462651s trace: https://jaeger.dev.storj.io/trace/73373c215464be3b FAIL (2025-11-22 16:42:18 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 22 16:41:49.669209413 (320h37m48.220679874s) [upload=282381]: Starting upload Nov 22 16:41:49.669269467 (320h37m48.220739928s) [upload=282381]: Done starting upload Nov 22 16:41:49.669428691 (320h37m48.220899152s) [upload=282381]: Uploading segments... Nov 22 16:41:49.669491310 (320h37m48.220961761s) [upload=282381]: Got next segment. Inline: false Nov 22 16:41:49.669533520 (320h37m48.221003971s) [upload=282381]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 22 16:41:49.669543579 (320h37m48.22101405s) [upload=282381]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 22 16:41:50.245063880 (320h37m48.796534341s) [seg_pos={0 0}, upload=282381]: Begin upload segment... Nov 22 16:41:50.245111712 (320h37m48.796582173s) [seg_pos={0 0}, upload=282381]: Done begin upload segment. Nov 22 16:41:50.245193718 (320h37m48.796664179s) [noise=true , address=145.79.227.9:4365 , piece_id=SO5POI5NR77NPOPQYXPIYKC5ETT7HTWGVAP4NLBUEMBISZONBV4Q, seg_pos={0 0}, upload=282381]: Uploading piece... Nov 22 16:41:50.245216191 (320h37m48.796686652s) [noise=true , address=77.90.31.21:4878 , piece_id=3H2XE3HXTKMKKVZXFFTSKCB3IM4X74OXLAX7TJOF6ISDGSEO4REQ, seg_pos={0 0}, upload=282381]: Uploading piece... Nov 22 16:41:50.245864785 (320h37m48.797335256s) [noise=true , address=45.91.116.13:4154 , piece_id=PJ4ABV44OHMZDGOX7UQFUK3ENI5JTXLJ2LNKZO7A7RQPJD53WSFQ, seg_pos={0 0}, upload=282381]: Uploading piece... Nov 22 16:41:50.246394764 (320h37m48.797865235s) [noise=true , address=145.79.227.21:4474 , piece_id=FE2KMZIEQSYGCHRM7LCRMFDVOP35RSLTEW2LC3DVGCYT5HVEKPIA, seg_pos={0 0}, upload=282381]: Uploading piece... Nov 22 16:41:50.246736996 (320h37m48.798207447s) [noise=true , address=77.90.31.13:4627 , piece_id=VPMNLZJDHW27RXRUMVKA7HPLF6TXKAQLXCKQ2GDWVA2HWCE72WIA, seg_pos={0 0}, upload=282381]: Uploading piece... Nov 22 16:41:50.247026979 (320h37m48.79849743s) [noise=true , address=145.79.227.31:4799 , piece_id=AYXSVOS7ONCJ4W4VKMHYJ2YAJ54QYBGYFSUF2A4LANKOII5HQ7EA, seg_pos={0 0}, upload=282381]: Uploading piece... Nov 22 16:41:50.247332229 (320h37m48.798802691s) [noise=true , address=45.91.116.7:4009 , piece_id=VSTM2QNABJSQRYGB763T757ARHMCF2BIFI4RLNZ33HY65B264USA, seg_pos={0 0}, upload=282381]: Uploading piece... Nov 22 16:41:50.247652190 (320h37m48.799122641s) [noise=true , address=145.79.227.2:4113 , piece_id=QEPFSWYRVU2QI6N63UIGAEO6XNGAY3N4LFGK23QDBMQHO5WXYJAA, seg_pos={0 0}, upload=282381]: Uploading piece... Nov 22 16:41:50.247934046 (320h37m48.799404497s) [noise=true , address=145.79.227.14:4837 , piece_id=YUEXSNF3D6BTVVTTDLIYU53TMY44V5YJ6ARVGHLVNT6FPQ4JCXDA, seg_pos={0 0}, upload=282381]: Uploading piece... Nov 22 16:41:50.248316444 (320h37m48.799786905s) [noise=true , address=145.79.227.23:4545 , piece_id=NNWS5ZTGLI7NFQT2AULIK6CUUEUSFYR3GSEGFZDXDGHFOGH2OTZQ, seg_pos={0 0}, upload=282381]: Uploading piece... Nov 22 16:41:50.248789244 (320h37m48.800259695s) [noise=true , address=145.79.227.27:4690 , piece_id=3QNAIMKSFRF62T5OTZWOFSFXA5N6ZASLOUJVCC2UYIJV6GILITDQ, seg_pos={0 0}, upload=282381]: Uploading piece... Nov 22 16:41:50.249041134 (320h37m48.800511585s) [noise=true , address=45.91.116.9:4262 , piece_id=LERQ3223LFXZGWMNPXN5KANRIIGHURV2V2GSC6QTWYNSCQG37OJQ, seg_pos={0 0}, upload=282381]: Uploading piece... Nov 22 16:41:50.249170470 (320h37m48.800640921s) [noise=true , address=145.79.227.15:4875 , piece_id=LU2QSMICPBLNAKC7JZNAD7XBWPCVWPMPX3KIKX2V7P44X2XGXJLA, seg_pos={0 0}, upload=282381]: Uploading piece... Nov 22 16:41:50.249465162 (320h37m48.800935613s) [noise=true , address=77.90.31.10:4517 , piece_id=FZZW7RIQZE6P5J6OKUFCLK5X6SI67QGRHCPJVZPSJDQ6A5VBLFBA, seg_pos={0 0}, upload=282381]: Uploading piece... Nov 22 16:41:50.249550193 (320h37m48.801020654s) [noise=true , address=145.79.227.30:4761 , piece_id=GF525VXMQ25FPBILIKRRQ6X35HSSF73RICFKPN3ODS6IXASCQVJQ, seg_pos={0 0}, upload=282381]: Uploading piece... Nov 22 16:41:50.250091574 (320h37m48.801562026s) [noise=true , address=77.90.31.12:4592 , piece_id=X2D67ERUFKBRZR3ND74GB7OMZ2LGWWKYDWR63PTGH4YMNE5E6MDA, seg_pos={0 0}, upload=282381]: Uploading piece... Nov 22 16:41:51.914100898 (320h37m50.465571359s) [upload=282381]: Next returned nil segment Nov 22 16:41:51.914115246 (320h37m50.465585707s) [upload=282381]: Waiting for error group managing segments... Nov 22 16:41:52.302424073 (320h37m50.853894534s) [noise=true , address=45.91.116.7:4009 , piece_id=VSTM2QNABJSQRYGB763T757ARHMCF2BIFI4RLNZ33HY65B264USA, seg_pos={0 0}, upload=282381]: Done uploading piece. err: <nil> Nov 22 16:41:52.323996711 (320h37m50.875467172s) [noise=true , address=45.91.116.13:4154 , piece_id=PJ4ABV44OHMZDGOX7UQFUK3ENI5JTXLJ2LNKZO7A7RQPJD53WSFQ, seg_pos={0 0}, upload=282381]: Done uploading piece. err: <nil> Nov 22 16:41:52.366609934 (320h37m50.918080395s) [noise=true , address=77.90.31.13:4627 , piece_id=VPMNLZJDHW27RXRUMVKA7HPLF6TXKAQLXCKQ2GDWVA2HWCE72WIA, seg_pos={0 0}, upload=282381]: Done uploading piece. err: <nil> Nov 22 16:41:52.383661573 (320h37m50.935132035s) [noise=true , address=77.90.31.10:4517 , piece_id=FZZW7RIQZE6P5J6OKUFCLK5X6SI67QGRHCPJVZPSJDQ6A5VBLFBA, seg_pos={0 0}, upload=282381]: Done uploading piece. err: <nil> Nov 22 16:41:52.921072257 (320h37m51.472542718s) [noise=true , address=77.90.31.12:4592 , piece_id=X2D67ERUFKBRZR3ND74GB7OMZ2LGWWKYDWR63PTGH4YMNE5E6MDA, seg_pos={0 0}, upload=282381]: Done uploading piece. err: <nil> Nov 22 16:41:53.072975457 (320h37m51.624445918s) [noise=true , address=77.90.31.21:4878 , piece_id=3H2XE3HXTKMKKVZXFFTSKCB3IM4X74OXLAX7TJOF6ISDGSEO4REQ, seg_pos={0 0}, upload=282381]: Done uploading piece. err: <nil> Nov 22 16:41:53.919460602 (320h37m52.470931063s) [noise=true , address=45.91.116.9:4262 , piece_id=LERQ3223LFXZGWMNPXN5KANRIIGHURV2V2GSC6QTWYNSCQG37OJQ, seg_pos={0 0}, upload=282381]: Done uploading piece. err: <nil> Nov 22 16:41:59.286028452 (320h37m57.837498913s) [noise=true , address=145.79.227.15:4875 , piece_id=LU2QSMICPBLNAKC7JZNAD7XBWPCVWPMPX3KIKX2V7P44X2XGXJLA, seg_pos={0 0}, upload=282381]: Done uploading piece. err: <nil> Nov 22 16:42:00.186876102 (320h37m58.738346563s) [noise=true , address=145.79.227.21:4474 , piece_id=FE2KMZIEQSYGCHRM7LCRMFDVOP35RSLTEW2LC3DVGCYT5HVEKPIA, seg_pos={0 0}, upload=282381]: Done uploading piece. err: <nil> Nov 22 16:42:02.278437229 (320h38m0.82990769s) [noise=true , address=145.79.227.30:4761 , piece_id=GF525VXMQ25FPBILIKRRQ6X35HSSF73RICFKPN3ODS6IXASCQVJQ, seg_pos={0 0}, upload=282381]: Done uploading piece. err: <nil> Nov 22 16:42:04.727839046 (320h38m3.279309507s) [noise=true , address=145.79.227.2:4113 , piece_id=QEPFSWYRVU2QI6N63UIGAEO6XNGAY3N4LFGK23QDBMQHO5WXYJAA, seg_pos={0 0}, upload=282381]: Done uploading piece. err: <nil> Nov 22 16:42:05.006455109 (320h38m3.557925571s) [noise=true , address=145.79.227.14:4837 , piece_id=YUEXSNF3D6BTVVTTDLIYU53TMY44V5YJ6ARVGHLVNT6FPQ4JCXDA, seg_pos={0 0}, upload=282381]: Done uploading piece. err: <nil> Nov 22 16:42:18.025379977 (320h38m16.576850438s) [noise=true , address=145.79.227.23:4545 , piece_id=NNWS5ZTGLI7NFQT2AULIK6CUUEUSFYR3GSEGFZDXDGHFOGH2OTZQ, seg_pos={0 0}, upload=282381]: Done uploading piece. err: ecclient: upload failed (node:12P5cnEza1Xxwg4qo1nau11zYHsHxqEAwhFsnt2UE1RhtTrJmra, address:145.79.227.23:4545): protocol: write tcp 188.166.225.170:40154->145.79.227.23:4545: use of closed network connection; write tcp 188.166.225.170:40154->145.79.227.23:4545: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40154->145.79.227.23:4545: use of closed network connection Nov 22 16:42:18.026301482 (320h38m16.577771934s) [noise=true , address=145.79.227.31:4799 , piece_id=AYXSVOS7ONCJ4W4VKMHYJ2YAJ54QYBGYFSUF2A4LANKOII5HQ7EA, seg_pos={0 0}, upload=282381]: Done uploading piece. err: ecclient: upload failed (node:1vMQm9RufCEcHca9k1JEhyED9MfYjvZcT2MFYic3PmEKyUoC7X, address:145.79.227.31:4799): protocol: read tcp 188.166.225.170:42764->145.79.227.31:4799: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 16:42:18.026393338 (320h38m16.577863789s) [noise=true , address=145.79.227.27:4690 , piece_id=3QNAIMKSFRF62T5OTZWOFSFXA5N6ZASLOUJVCC2UYIJV6GILITDQ, seg_pos={0 0}, upload=282381]: Done uploading piece. err: ecclient: upload failed (node:12fyCro9gG2d3myk6Cju5vmf4CbkqbJN2PKAP8EbpcT1Rtao27H, address:145.79.227.27:4690): protocol: read tcp 188.166.225.170:43594->145.79.227.27:4690: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 16:42:18.026594139 (320h38m16.57806461s) [noise=true , address=145.79.227.9:4365 , piece_id=SO5POI5NR77NPOPQYXPIYKC5ETT7HTWGVAP4NLBUEMBISZONBV4Q, seg_pos={0 0}, upload=282381]: Done uploading piece. err: ecclient: upload failed (node:12rvMymD9ejLTfLTfeF4VAycp2SdGUMBtR3o45x8JeMkCDWpCHq, address:145.79.227.9:4365): protocol: read tcp 188.166.225.170:56822->145.79.227.9:4365: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 16:42:18.026628976 (320h38m16.578099437s) [seg_pos={0 0}, upload=282381]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 22 16:42:18.509068735 (320h38m17.060539196s) [upload=282381]: Done uploading segments... Nov 22 16:42:18.509116086 (320h38m17.060586547s) [upload=282381]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 30.486478491s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 14 out of 60 is failed (threshold=6) [XX.XX.......X..XX......X.X...X.......X...........X.X....X...] duration: 3.492556526s trace: https://jaeger.dev.storj.io/trace/102f7d28bf9f3520 FAIL (2025-11-22 16:42:06 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.492587505s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [......................XX....................................] duration: 15.258038341s trace: https://jaeger.dev.storj.io/trace/45889407e3b21f30 FAIL (2025-11-22 16:41:41 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ad Nov 22 16:15:10.157378989 (320h11m8.70884945s) [upload=281977]: Uploading segments... Nov 22 16:15:10.157518405 (320h11m8.708988866s) [upload=281977]: Got next segment. Inline: false Nov 22 16:15:10.157568901 (320h11m8.709039373s) [upload=281977]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 22 16:15:10.157576005 (320h11m8.709046466s) [upload=281977]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 22 16:15:10.422893065 (320h11m8.974363526s) [seg_pos={0 0}, upload=281977]: Begin upload segment... Nov 22 16:15:10.422954472 (320h11m8.974424933s) [seg_pos={0 0}, upload=281977]: Done begin upload segment. Nov 22 16:15:10.422989378 (320h11m8.974459839s) [noise=true , address=77.90.31.12:4589 , piece_id=DMJXC2AGYUS6IJDU57YMRVQ4XITS6C65KQPKPK77BAVNIWEDYW6A, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.423082535 (320h11m8.974553026s) [noise=true , address=145.79.227.11:4731 , piece_id=ASYJFIYGHG2VMQE5Y2W6J2MH4HE4TJX5HB36VDPNL6JI6LOBOWMA, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.423840959 (320h11m8.97531142s) [noise=true , address=45.91.116.4:4940 , piece_id=G7JVOOU24MGUUAJST54AWGAHJCQ3NEEDNYJCT7274VL3PMXXYEEA, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.424298210 (320h11m8.975768671s) [noise=true , address=145.79.227.28:4725 , piece_id=WAPK2NYLOGLXI4BOM6LA43UNUYKKVGCD6LA2KKPPIMXROR5AHCEA, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.424920144 (320h11m8.976390615s) [noise=true , address=145.79.227.14:4840 , piece_id=IFKFP5EBBMSNGDII2I2MT5IHK223AP2LD2RZLGMI2UFH7EVB6R3A, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.425382976 (320h11m8.976853437s) [noise=true , address=77.90.31.15:4699 , piece_id=6Q26DE27DV2TCYPSVMG64R36I4NJFZ6XMSX6ISYYJI6XD3E2TKQA, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.425904149 (320h11m8.97737461s) [noise=true , address=145.79.227.22:4509 , piece_id=XBHW2Q4ZMTXHM3DXV2L3H3UJRHW5NYMLO2Z5WMBPSR3QEKWM4YYA, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.426380896 (320h11m8.977851367s) [noise=true , address=145.79.227.13:4802 , piece_id=VQ25RRLCCHN2BIEARHDMGO6JXOLM3RAAYSBGDJRZZAA7LU6XIORA, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.427020925 (320h11m8.978491386s) [noise=true , address=145.79.227.21:4474 , piece_id=PECKNOFRUZZIVY5XJ32WCP3CFBJ7LERMD6QFZZPLAXA3B47VNYBQ, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.428223965 (320h11m8.979694426s) [noise=true , address=145.79.227.8:4331 , piece_id=IXCEZX2MCTDJJWMQY67WUQDLVKUBC6ILSZ66OTDRGZCF43DPPGSQ, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.430197163 (320h11m8.981667624s) [noise=true , address=77.90.31.12:4590 , piece_id=TUKND5MYHBQIH5FKBG6KIFDNAEUAGEX4SZ62D2BJFA77U2KU5UKQ, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.432639544 (320h11m8.984109995s) [noise=true , address=77.90.31.18:4807 , piece_id=QDVLUO3SEXUQJCCMJ4O5VNQK2XBTLXTYKKNZXDFTLXBFRUHMEKEQ, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.435162708 (320h11m8.986633169s) [noise=true , address=145.79.227.15:4874 , piece_id=FE5ZYSKFAKRZZ6RVLVKGZN46PBS5BZFZP5PKVYZNZFBXYGFIKFKA, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.437291813 (320h11m8.988762274s) [noise=true , address=145.79.227.6:4259 , piece_id=JFLTPOWA63S22QGQ6LN4WRFYVV2XQIDRLL7VGWFJRMCPXK5M6MXA, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.439344221 (320h11m8.990814682s) [noise=true , address=77.90.31.19:4841 , piece_id=VKYGMN3H4X2JJUFLNDGZZSCV5HCLQRLR53X5J36FGQ565MKWMNCA, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.440379501 (320h11m8.991849973s) [noise=true , address=145.79.227.20:4437 , piece_id=6ION5L5UGDHAJQTYOAC34BTWDUJOJQXDF4XBRF7SQMRY57BYECOQ, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:12.150193113 (320h11m10.701663574s) [upload=281977]: Next returned nil segment Nov 22 16:15:12.150240513 (320h11m10.701710974s) [upload=281977]: Waiting for error group managing segments... Nov 22 16:15:12.562473597 (320h11m11.113944048s) [noise=true , address=77.90.31.15:4699 , piece_id=6Q26DE27DV2TCYPSVMG64R36I4NJFZ6XMSX6ISYYJI6XD3E2TKQA, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:12.579741378 (320h11m11.131211839s) [noise=true , address=77.90.31.12:4590 , piece_id=TUKND5MYHBQIH5FKBG6KIFDNAEUAGEX4SZ62D2BJFA77U2KU5UKQ, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:12.586731719 (320h11m11.13820218s) [noise=true , address=45.91.116.4:4940 , piece_id=G7JVOOU24MGUUAJST54AWGAHJCQ3NEEDNYJCT7274VL3PMXXYEEA, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:12.598257075 (320h11m11.149727536s) [noise=true , address=77.90.31.12:4589 , piece_id=DMJXC2AGYUS6IJDU57YMRVQ4XITS6C65KQPKPK77BAVNIWEDYW6A, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:12.622577225 (320h11m11.174047686s) [noise=true , address=77.90.31.19:4841 , piece_id=VKYGMN3H4X2JJUFLNDGZZSCV5HCLQRLR53X5J36FGQ565MKWMNCA, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:12.754772838 (320h11m11.306243299s) [noise=true , address=77.90.31.18:4807 , piece_id=QDVLUO3SEXUQJCCMJ4O5VNQK2XBTLXTYKKNZXDFTLXBFRUHMEKEQ, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:19.382583530 (320h11m17.934053981s) [noise=true , address=145.79.227.15:4874 , piece_id=FE5ZYSKFAKRZZ6RVLVKGZN46PBS5BZFZP5PKVYZNZFBXYGFIKFKA, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:21.514760656 (320h11m20.066231117s) [noise=true , address=145.79.227.22:4509 , piece_id=XBHW2Q4ZMTXHM3DXV2L3H3UJRHW5NYMLO2Z5WMBPSR3QEKWM4YYA, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:23.636942539 (320h11m22.188413s) [noise=true , address=145.79.227.6:4259 , piece_id=JFLTPOWA63S22QGQ6LN4WRFYVV2XQIDRLL7VGWFJRMCPXK5M6MXA, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:23.872352010 (320h11m22.423822471s) [noise=true , address=145.79.227.28:4725 , piece_id=WAPK2NYLOGLXI4BOM6LA43UNUYKKVGCD6LA2KKPPIMXROR5AHCEA, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:25.966169160 (320h11m24.517639612s) [noise=true , address=145.79.227.8:4331 , piece_id=IXCEZX2MCTDJJWMQY67WUQDLVKUBC6ILSZ66OTDRGZCF43DPPGSQ, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:26.960865746 (320h11m25.512336207s) [noise=true , address=145.79.227.14:4840 , piece_id=IFKFP5EBBMSNGDII2I2MT5IHK223AP2LD2RZLGMI2UFH7EVB6R3A, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:36.322690505 (320h11m34.874160966s) [noise=true , address=145.79.227.21:4474 , piece_id=PECKNOFRUZZIVY5XJ32WCP3CFBJ7LERMD6QFZZPLAXA3B47VNYBQ, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:36.322718137 (320h11m34.874188599s) [seg_pos={0 0}, upload=281977]: Segment reached optimal threshold of 13 pieces. Nov 22 16:15:36.323447516 (320h11m34.874917977s) [noise=true , address=145.79.227.11:4731 , piece_id=ASYJFIYGHG2VMQE5Y2W6J2MH4HE4TJX5HB36VDPNL6JI6LOBOWMA, seg_pos={0 0}, upload=281977]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9): protocol: rpc: dial tcp 145.79.227.11:4731: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 16:15:36.323459900 (320h11m34.874930371s) [noise=true , address=145.79.227.11:4731 , piece_id=ASYJFIYGHG2VMQE5Y2W6J2MH4HE4TJX5HB36VDPNL6JI6LOBOWMA, seg_pos={0 0}, upload=281977]: Long tail cancellation due to optimal threshold Nov 22 16:15:36.323615366 (320h11m34.875085817s) [noise=true , address=145.79.227.20:4437 , piece_id=6ION5L5UGDHAJQTYOAC34BTWDUJOJQXDF4XBRF7SQMRY57BYECOQ, seg_pos={0 0}, upload=281977]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1G3fe4SHMdNAjAmqLNyFb6qDKL94aAWGYE9qhczKmozKFhKKEV): protocol: write tcp 188.166.225.170:49846->145.79.227.20:4437: use of closed network connection; write tcp 188.166.225.170:49846->145.79.227.20:4437: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49846->145.79.227.20:4437: use of closed network connection Nov 22 16:15:36.323624343 (320h11m34.875094804s) [noise=true , address=145.79.227.20:4437 , piece_id=6ION5L5UGDHAJQTYOAC34BTWDUJOJQXDF4XBRF7SQMRY57BYECOQ, seg_pos={0 0}, upload=281977]: Long tail cancellation due to optimal threshold Nov 22 16:15:36.323867195 (320h11m34.875337677s) [noise=true , address=145.79.227.13:4802 , piece_id=VQ25RRLCCHN2BIEARHDMGO6JXOLM3RAAYSBGDJRZZAA7LU6XIORA, seg_pos={0 0}, upload=281977]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4): protocol: read tcp 188.166.225.170:59064->145.79.227.13:4802: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 16:15:36.323877184 (320h11m34.875347645s) [noise=true , address=145.79.227.13:4802 , piece_id=VQ25RRLCCHN2BIEARHDMGO6JXOLM3RAAYSBGDJRZZAA7LU6XIORA, seg_pos={0 0}, upload=281977]: Long tail cancellation due to optimal threshold Nov 22 16:15:36.323899958 (320h11m34.875370419s) [seg_pos={0 0}, upload=281977]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 22 16:15:36.324029074 (320h11m34.875499535s) [upload=281977]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 22 16:15:36.324036789 (320h11m34.87550726s) [upload=281977]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 22 16:15:36.620020293 (320h11m35.171490744s) [upload=281977]: Done uploading segments... Nov 22 16:15:36.620042956 (320h11m35.171513417s) [upload=281977]: Upload finished. err: <nil> Duration: 15.258093776s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................X...........................................] duration: 19.158074499s trace: https://jaeger.dev.storj.io/trace/13bcd3d57614c8f8 FAIL (2025-11-22 16:41:28 UTC) orbiter warning: context deadline exceeded Duration: 19.158109725s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 15 out of 60 is failed (threshold=6) [XXX.XX.......X..XX......X.X...X.......X...........X.X....X..] duration: 3.542090036s trace: https://jaeger.dev.storj.io/trace/102f7d28bf9f3520 FAIL (2025-11-22 16:40:59 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.542119953s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.......................XX...................................] duration: 27.070335077s trace: https://jaeger.dev.storj.io/trace/45889407e3b21f30 FAIL (2025-11-22 16:40:43 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ad Nov 22 16:15:10.157378989 (320h11m8.70884945s) [upload=281977]: Uploading segments... Nov 22 16:15:10.157518405 (320h11m8.708988866s) [upload=281977]: Got next segment. Inline: false Nov 22 16:15:10.157568901 (320h11m8.709039373s) [upload=281977]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 22 16:15:10.157576005 (320h11m8.709046466s) [upload=281977]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 22 16:15:10.422893065 (320h11m8.974363526s) [seg_pos={0 0}, upload=281977]: Begin upload segment... Nov 22 16:15:10.422954472 (320h11m8.974424933s) [seg_pos={0 0}, upload=281977]: Done begin upload segment. Nov 22 16:15:10.422989378 (320h11m8.974459839s) [noise=true , address=77.90.31.12:4589 , piece_id=DMJXC2AGYUS6IJDU57YMRVQ4XITS6C65KQPKPK77BAVNIWEDYW6A, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.423082535 (320h11m8.974553026s) [noise=true , address=145.79.227.11:4731 , piece_id=ASYJFIYGHG2VMQE5Y2W6J2MH4HE4TJX5HB36VDPNL6JI6LOBOWMA, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.423840959 (320h11m8.97531142s) [noise=true , address=45.91.116.4:4940 , piece_id=G7JVOOU24MGUUAJST54AWGAHJCQ3NEEDNYJCT7274VL3PMXXYEEA, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.424298210 (320h11m8.975768671s) [noise=true , address=145.79.227.28:4725 , piece_id=WAPK2NYLOGLXI4BOM6LA43UNUYKKVGCD6LA2KKPPIMXROR5AHCEA, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.424920144 (320h11m8.976390615s) [noise=true , address=145.79.227.14:4840 , piece_id=IFKFP5EBBMSNGDII2I2MT5IHK223AP2LD2RZLGMI2UFH7EVB6R3A, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.425382976 (320h11m8.976853437s) [noise=true , address=77.90.31.15:4699 , piece_id=6Q26DE27DV2TCYPSVMG64R36I4NJFZ6XMSX6ISYYJI6XD3E2TKQA, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.425904149 (320h11m8.97737461s) [noise=true , address=145.79.227.22:4509 , piece_id=XBHW2Q4ZMTXHM3DXV2L3H3UJRHW5NYMLO2Z5WMBPSR3QEKWM4YYA, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.426380896 (320h11m8.977851367s) [noise=true , address=145.79.227.13:4802 , piece_id=VQ25RRLCCHN2BIEARHDMGO6JXOLM3RAAYSBGDJRZZAA7LU6XIORA, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.427020925 (320h11m8.978491386s) [noise=true , address=145.79.227.21:4474 , piece_id=PECKNOFRUZZIVY5XJ32WCP3CFBJ7LERMD6QFZZPLAXA3B47VNYBQ, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.428223965 (320h11m8.979694426s) [noise=true , address=145.79.227.8:4331 , piece_id=IXCEZX2MCTDJJWMQY67WUQDLVKUBC6ILSZ66OTDRGZCF43DPPGSQ, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.430197163 (320h11m8.981667624s) [noise=true , address=77.90.31.12:4590 , piece_id=TUKND5MYHBQIH5FKBG6KIFDNAEUAGEX4SZ62D2BJFA77U2KU5UKQ, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.432639544 (320h11m8.984109995s) [noise=true , address=77.90.31.18:4807 , piece_id=QDVLUO3SEXUQJCCMJ4O5VNQK2XBTLXTYKKNZXDFTLXBFRUHMEKEQ, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.435162708 (320h11m8.986633169s) [noise=true , address=145.79.227.15:4874 , piece_id=FE5ZYSKFAKRZZ6RVLVKGZN46PBS5BZFZP5PKVYZNZFBXYGFIKFKA, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.437291813 (320h11m8.988762274s) [noise=true , address=145.79.227.6:4259 , piece_id=JFLTPOWA63S22QGQ6LN4WRFYVV2XQIDRLL7VGWFJRMCPXK5M6MXA, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.439344221 (320h11m8.990814682s) [noise=true , address=77.90.31.19:4841 , piece_id=VKYGMN3H4X2JJUFLNDGZZSCV5HCLQRLR53X5J36FGQ565MKWMNCA, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.440379501 (320h11m8.991849973s) [noise=true , address=145.79.227.20:4437 , piece_id=6ION5L5UGDHAJQTYOAC34BTWDUJOJQXDF4XBRF7SQMRY57BYECOQ, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:12.150193113 (320h11m10.701663574s) [upload=281977]: Next returned nil segment Nov 22 16:15:12.150240513 (320h11m10.701710974s) [upload=281977]: Waiting for error group managing segments... Nov 22 16:15:12.562473597 (320h11m11.113944048s) [noise=true , address=77.90.31.15:4699 , piece_id=6Q26DE27DV2TCYPSVMG64R36I4NJFZ6XMSX6ISYYJI6XD3E2TKQA, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:12.579741378 (320h11m11.131211839s) [noise=true , address=77.90.31.12:4590 , piece_id=TUKND5MYHBQIH5FKBG6KIFDNAEUAGEX4SZ62D2BJFA77U2KU5UKQ, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:12.586731719 (320h11m11.13820218s) [noise=true , address=45.91.116.4:4940 , piece_id=G7JVOOU24MGUUAJST54AWGAHJCQ3NEEDNYJCT7274VL3PMXXYEEA, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:12.598257075 (320h11m11.149727536s) [noise=true , address=77.90.31.12:4589 , piece_id=DMJXC2AGYUS6IJDU57YMRVQ4XITS6C65KQPKPK77BAVNIWEDYW6A, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:12.622577225 (320h11m11.174047686s) [noise=true , address=77.90.31.19:4841 , piece_id=VKYGMN3H4X2JJUFLNDGZZSCV5HCLQRLR53X5J36FGQ565MKWMNCA, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:12.754772838 (320h11m11.306243299s) [noise=true , address=77.90.31.18:4807 , piece_id=QDVLUO3SEXUQJCCMJ4O5VNQK2XBTLXTYKKNZXDFTLXBFRUHMEKEQ, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:19.382583530 (320h11m17.934053981s) [noise=true , address=145.79.227.15:4874 , piece_id=FE5ZYSKFAKRZZ6RVLVKGZN46PBS5BZFZP5PKVYZNZFBXYGFIKFKA, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:21.514760656 (320h11m20.066231117s) [noise=true , address=145.79.227.22:4509 , piece_id=XBHW2Q4ZMTXHM3DXV2L3H3UJRHW5NYMLO2Z5WMBPSR3QEKWM4YYA, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:23.636942539 (320h11m22.188413s) [noise=true , address=145.79.227.6:4259 , piece_id=JFLTPOWA63S22QGQ6LN4WRFYVV2XQIDRLL7VGWFJRMCPXK5M6MXA, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:23.872352010 (320h11m22.423822471s) [noise=true , address=145.79.227.28:4725 , piece_id=WAPK2NYLOGLXI4BOM6LA43UNUYKKVGCD6LA2KKPPIMXROR5AHCEA, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:25.966169160 (320h11m24.517639612s) [noise=true , address=145.79.227.8:4331 , piece_id=IXCEZX2MCTDJJWMQY67WUQDLVKUBC6ILSZ66OTDRGZCF43DPPGSQ, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:26.960865746 (320h11m25.512336207s) [noise=true , address=145.79.227.14:4840 , piece_id=IFKFP5EBBMSNGDII2I2MT5IHK223AP2LD2RZLGMI2UFH7EVB6R3A, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:36.322690505 (320h11m34.874160966s) [noise=true , address=145.79.227.21:4474 , piece_id=PECKNOFRUZZIVY5XJ32WCP3CFBJ7LERMD6QFZZPLAXA3B47VNYBQ, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:36.322718137 (320h11m34.874188599s) [seg_pos={0 0}, upload=281977]: Segment reached optimal threshold of 13 pieces. Nov 22 16:15:36.323447516 (320h11m34.874917977s) [noise=true , address=145.79.227.11:4731 , piece_id=ASYJFIYGHG2VMQE5Y2W6J2MH4HE4TJX5HB36VDPNL6JI6LOBOWMA, seg_pos={0 0}, upload=281977]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9): protocol: rpc: dial tcp 145.79.227.11:4731: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 16:15:36.323459900 (320h11m34.874930371s) [noise=true , address=145.79.227.11:4731 , piece_id=ASYJFIYGHG2VMQE5Y2W6J2MH4HE4TJX5HB36VDPNL6JI6LOBOWMA, seg_pos={0 0}, upload=281977]: Long tail cancellation due to optimal threshold Nov 22 16:15:36.323615366 (320h11m34.875085817s) [noise=true , address=145.79.227.20:4437 , piece_id=6ION5L5UGDHAJQTYOAC34BTWDUJOJQXDF4XBRF7SQMRY57BYECOQ, seg_pos={0 0}, upload=281977]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1G3fe4SHMdNAjAmqLNyFb6qDKL94aAWGYE9qhczKmozKFhKKEV): protocol: write tcp 188.166.225.170:49846->145.79.227.20:4437: use of closed network connection; write tcp 188.166.225.170:49846->145.79.227.20:4437: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49846->145.79.227.20:4437: use of closed network connection Nov 22 16:15:36.323624343 (320h11m34.875094804s) [noise=true , address=145.79.227.20:4437 , piece_id=6ION5L5UGDHAJQTYOAC34BTWDUJOJQXDF4XBRF7SQMRY57BYECOQ, seg_pos={0 0}, upload=281977]: Long tail cancellation due to optimal threshold Nov 22 16:15:36.323867195 (320h11m34.875337677s) [noise=true , address=145.79.227.13:4802 , piece_id=VQ25RRLCCHN2BIEARHDMGO6JXOLM3RAAYSBGDJRZZAA7LU6XIORA, seg_pos={0 0}, upload=281977]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4): protocol: read tcp 188.166.225.170:59064->145.79.227.13:4802: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 16:15:36.323877184 (320h11m34.875347645s) [noise=true , address=145.79.227.13:4802 , piece_id=VQ25RRLCCHN2BIEARHDMGO6JXOLM3RAAYSBGDJRZZAA7LU6XIORA, seg_pos={0 0}, upload=281977]: Long tail cancellation due to optimal threshold Nov 22 16:15:36.323899958 (320h11m34.875370419s) [seg_pos={0 0}, upload=281977]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 22 16:15:36.324029074 (320h11m34.875499535s) [upload=281977]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 22 16:15:36.324036789 (320h11m34.87550726s) [upload=281977]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 22 16:15:36.620020293 (320h11m35.171490744s) [upload=281977]: Done uploading segments... Nov 22 16:15:36.620042956 (320h11m35.171513417s) [upload=281977]: Upload finished. err: <nil> Duration: 27.070366447s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................X..........................................] duration: 16.573142307s trace: https://jaeger.dev.storj.io/trace/13bcd3d57614c8f8 FAIL (2025-11-22 16:40:37 UTC) orbiter warning: context deadline exceeded Duration: 16.573171612s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................X.........................................] duration: 10.777741875s trace: https://jaeger.dev.storj.io/trace/13bcd3d57614c8f8 FAIL (2025-11-22 16:39:56 UTC) orbiter warning: context deadline exceeded Duration: 10.777772033s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 15 out of 60 is failed (threshold=6) [.XXX.XX.......X..XX......X.X...X.......X...........X.X....X.] duration: 3.671863858s trace: https://jaeger.dev.storj.io/trace/102f7d28bf9f3520 FAIL (2025-11-22 16:39:47 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.671896239s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [........................XX..................................] duration: 12.478835364s trace: https://jaeger.dev.storj.io/trace/45889407e3b21f30 FAIL (2025-11-22 16:39:44 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ad Nov 22 16:15:10.157378989 (320h11m8.70884945s) [upload=281977]: Uploading segments... Nov 22 16:15:10.157518405 (320h11m8.708988866s) [upload=281977]: Got next segment. Inline: false Nov 22 16:15:10.157568901 (320h11m8.709039373s) [upload=281977]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 22 16:15:10.157576005 (320h11m8.709046466s) [upload=281977]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 22 16:15:10.422893065 (320h11m8.974363526s) [seg_pos={0 0}, upload=281977]: Begin upload segment... Nov 22 16:15:10.422954472 (320h11m8.974424933s) [seg_pos={0 0}, upload=281977]: Done begin upload segment. Nov 22 16:15:10.422989378 (320h11m8.974459839s) [noise=true , address=77.90.31.12:4589 , piece_id=DMJXC2AGYUS6IJDU57YMRVQ4XITS6C65KQPKPK77BAVNIWEDYW6A, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.423082535 (320h11m8.974553026s) [noise=true , address=145.79.227.11:4731 , piece_id=ASYJFIYGHG2VMQE5Y2W6J2MH4HE4TJX5HB36VDPNL6JI6LOBOWMA, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.423840959 (320h11m8.97531142s) [noise=true , address=45.91.116.4:4940 , piece_id=G7JVOOU24MGUUAJST54AWGAHJCQ3NEEDNYJCT7274VL3PMXXYEEA, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.424298210 (320h11m8.975768671s) [noise=true , address=145.79.227.28:4725 , piece_id=WAPK2NYLOGLXI4BOM6LA43UNUYKKVGCD6LA2KKPPIMXROR5AHCEA, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.424920144 (320h11m8.976390615s) [noise=true , address=145.79.227.14:4840 , piece_id=IFKFP5EBBMSNGDII2I2MT5IHK223AP2LD2RZLGMI2UFH7EVB6R3A, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.425382976 (320h11m8.976853437s) [noise=true , address=77.90.31.15:4699 , piece_id=6Q26DE27DV2TCYPSVMG64R36I4NJFZ6XMSX6ISYYJI6XD3E2TKQA, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.425904149 (320h11m8.97737461s) [noise=true , address=145.79.227.22:4509 , piece_id=XBHW2Q4ZMTXHM3DXV2L3H3UJRHW5NYMLO2Z5WMBPSR3QEKWM4YYA, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.426380896 (320h11m8.977851367s) [noise=true , address=145.79.227.13:4802 , piece_id=VQ25RRLCCHN2BIEARHDMGO6JXOLM3RAAYSBGDJRZZAA7LU6XIORA, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.427020925 (320h11m8.978491386s) [noise=true , address=145.79.227.21:4474 , piece_id=PECKNOFRUZZIVY5XJ32WCP3CFBJ7LERMD6QFZZPLAXA3B47VNYBQ, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.428223965 (320h11m8.979694426s) [noise=true , address=145.79.227.8:4331 , piece_id=IXCEZX2MCTDJJWMQY67WUQDLVKUBC6ILSZ66OTDRGZCF43DPPGSQ, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.430197163 (320h11m8.981667624s) [noise=true , address=77.90.31.12:4590 , piece_id=TUKND5MYHBQIH5FKBG6KIFDNAEUAGEX4SZ62D2BJFA77U2KU5UKQ, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.432639544 (320h11m8.984109995s) [noise=true , address=77.90.31.18:4807 , piece_id=QDVLUO3SEXUQJCCMJ4O5VNQK2XBTLXTYKKNZXDFTLXBFRUHMEKEQ, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.435162708 (320h11m8.986633169s) [noise=true , address=145.79.227.15:4874 , piece_id=FE5ZYSKFAKRZZ6RVLVKGZN46PBS5BZFZP5PKVYZNZFBXYGFIKFKA, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.437291813 (320h11m8.988762274s) [noise=true , address=145.79.227.6:4259 , piece_id=JFLTPOWA63S22QGQ6LN4WRFYVV2XQIDRLL7VGWFJRMCPXK5M6MXA, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.439344221 (320h11m8.990814682s) [noise=true , address=77.90.31.19:4841 , piece_id=VKYGMN3H4X2JJUFLNDGZZSCV5HCLQRLR53X5J36FGQ565MKWMNCA, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.440379501 (320h11m8.991849973s) [noise=true , address=145.79.227.20:4437 , piece_id=6ION5L5UGDHAJQTYOAC34BTWDUJOJQXDF4XBRF7SQMRY57BYECOQ, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:12.150193113 (320h11m10.701663574s) [upload=281977]: Next returned nil segment Nov 22 16:15:12.150240513 (320h11m10.701710974s) [upload=281977]: Waiting for error group managing segments... Nov 22 16:15:12.562473597 (320h11m11.113944048s) [noise=true , address=77.90.31.15:4699 , piece_id=6Q26DE27DV2TCYPSVMG64R36I4NJFZ6XMSX6ISYYJI6XD3E2TKQA, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:12.579741378 (320h11m11.131211839s) [noise=true , address=77.90.31.12:4590 , piece_id=TUKND5MYHBQIH5FKBG6KIFDNAEUAGEX4SZ62D2BJFA77U2KU5UKQ, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:12.586731719 (320h11m11.13820218s) [noise=true , address=45.91.116.4:4940 , piece_id=G7JVOOU24MGUUAJST54AWGAHJCQ3NEEDNYJCT7274VL3PMXXYEEA, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:12.598257075 (320h11m11.149727536s) [noise=true , address=77.90.31.12:4589 , piece_id=DMJXC2AGYUS6IJDU57YMRVQ4XITS6C65KQPKPK77BAVNIWEDYW6A, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:12.622577225 (320h11m11.174047686s) [noise=true , address=77.90.31.19:4841 , piece_id=VKYGMN3H4X2JJUFLNDGZZSCV5HCLQRLR53X5J36FGQ565MKWMNCA, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:12.754772838 (320h11m11.306243299s) [noise=true , address=77.90.31.18:4807 , piece_id=QDVLUO3SEXUQJCCMJ4O5VNQK2XBTLXTYKKNZXDFTLXBFRUHMEKEQ, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:19.382583530 (320h11m17.934053981s) [noise=true , address=145.79.227.15:4874 , piece_id=FE5ZYSKFAKRZZ6RVLVKGZN46PBS5BZFZP5PKVYZNZFBXYGFIKFKA, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:21.514760656 (320h11m20.066231117s) [noise=true , address=145.79.227.22:4509 , piece_id=XBHW2Q4ZMTXHM3DXV2L3H3UJRHW5NYMLO2Z5WMBPSR3QEKWM4YYA, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:23.636942539 (320h11m22.188413s) [noise=true , address=145.79.227.6:4259 , piece_id=JFLTPOWA63S22QGQ6LN4WRFYVV2XQIDRLL7VGWFJRMCPXK5M6MXA, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:23.872352010 (320h11m22.423822471s) [noise=true , address=145.79.227.28:4725 , piece_id=WAPK2NYLOGLXI4BOM6LA43UNUYKKVGCD6LA2KKPPIMXROR5AHCEA, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:25.966169160 (320h11m24.517639612s) [noise=true , address=145.79.227.8:4331 , piece_id=IXCEZX2MCTDJJWMQY67WUQDLVKUBC6ILSZ66OTDRGZCF43DPPGSQ, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:26.960865746 (320h11m25.512336207s) [noise=true , address=145.79.227.14:4840 , piece_id=IFKFP5EBBMSNGDII2I2MT5IHK223AP2LD2RZLGMI2UFH7EVB6R3A, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:36.322690505 (320h11m34.874160966s) [noise=true , address=145.79.227.21:4474 , piece_id=PECKNOFRUZZIVY5XJ32WCP3CFBJ7LERMD6QFZZPLAXA3B47VNYBQ, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:36.322718137 (320h11m34.874188599s) [seg_pos={0 0}, upload=281977]: Segment reached optimal threshold of 13 pieces. Nov 22 16:15:36.323447516 (320h11m34.874917977s) [noise=true , address=145.79.227.11:4731 , piece_id=ASYJFIYGHG2VMQE5Y2W6J2MH4HE4TJX5HB36VDPNL6JI6LOBOWMA, seg_pos={0 0}, upload=281977]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9): protocol: rpc: dial tcp 145.79.227.11:4731: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 16:15:36.323459900 (320h11m34.874930371s) [noise=true , address=145.79.227.11:4731 , piece_id=ASYJFIYGHG2VMQE5Y2W6J2MH4HE4TJX5HB36VDPNL6JI6LOBOWMA, seg_pos={0 0}, upload=281977]: Long tail cancellation due to optimal threshold Nov 22 16:15:36.323615366 (320h11m34.875085817s) [noise=true , address=145.79.227.20:4437 , piece_id=6ION5L5UGDHAJQTYOAC34BTWDUJOJQXDF4XBRF7SQMRY57BYECOQ, seg_pos={0 0}, upload=281977]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1G3fe4SHMdNAjAmqLNyFb6qDKL94aAWGYE9qhczKmozKFhKKEV): protocol: write tcp 188.166.225.170:49846->145.79.227.20:4437: use of closed network connection; write tcp 188.166.225.170:49846->145.79.227.20:4437: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49846->145.79.227.20:4437: use of closed network connection Nov 22 16:15:36.323624343 (320h11m34.875094804s) [noise=true , address=145.79.227.20:4437 , piece_id=6ION5L5UGDHAJQTYOAC34BTWDUJOJQXDF4XBRF7SQMRY57BYECOQ, seg_pos={0 0}, upload=281977]: Long tail cancellation due to optimal threshold Nov 22 16:15:36.323867195 (320h11m34.875337677s) [noise=true , address=145.79.227.13:4802 , piece_id=VQ25RRLCCHN2BIEARHDMGO6JXOLM3RAAYSBGDJRZZAA7LU6XIORA, seg_pos={0 0}, upload=281977]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4): protocol: read tcp 188.166.225.170:59064->145.79.227.13:4802: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 16:15:36.323877184 (320h11m34.875347645s) [noise=true , address=145.79.227.13:4802 , piece_id=VQ25RRLCCHN2BIEARHDMGO6JXOLM3RAAYSBGDJRZZAA7LU6XIORA, seg_pos={0 0}, upload=281977]: Long tail cancellation due to optimal threshold Nov 22 16:15:36.323899958 (320h11m34.875370419s) [seg_pos={0 0}, upload=281977]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 22 16:15:36.324029074 (320h11m34.875499535s) [upload=281977]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 22 16:15:36.324036789 (320h11m34.87550726s) [upload=281977]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 22 16:15:36.620020293 (320h11m35.171490744s) [upload=281977]: Done uploading segments... Nov 22 16:15:36.620042956 (320h11m35.171513417s) [upload=281977]: Upload finished. err: <nil> Duration: 12.478866925s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................X........................................] duration: 12.868195603s trace: https://jaeger.dev.storj.io/trace/13bcd3d57614c8f8 FAIL (2025-11-22 16:39:08 UTC) orbiter warning: context deadline exceeded Duration: 12.868221292s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.........................XX.................................] duration: 11.446219133s trace: https://jaeger.dev.storj.io/trace/45889407e3b21f30 FAIL (2025-11-22 16:38:39 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ad Nov 22 16:15:10.157378989 (320h11m8.70884945s) [upload=281977]: Uploading segments... Nov 22 16:15:10.157518405 (320h11m8.708988866s) [upload=281977]: Got next segment. Inline: false Nov 22 16:15:10.157568901 (320h11m8.709039373s) [upload=281977]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 22 16:15:10.157576005 (320h11m8.709046466s) [upload=281977]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 22 16:15:10.422893065 (320h11m8.974363526s) [seg_pos={0 0}, upload=281977]: Begin upload segment... Nov 22 16:15:10.422954472 (320h11m8.974424933s) [seg_pos={0 0}, upload=281977]: Done begin upload segment. Nov 22 16:15:10.422989378 (320h11m8.974459839s) [noise=true , address=77.90.31.12:4589 , piece_id=DMJXC2AGYUS6IJDU57YMRVQ4XITS6C65KQPKPK77BAVNIWEDYW6A, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.423082535 (320h11m8.974553026s) [noise=true , address=145.79.227.11:4731 , piece_id=ASYJFIYGHG2VMQE5Y2W6J2MH4HE4TJX5HB36VDPNL6JI6LOBOWMA, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.423840959 (320h11m8.97531142s) [noise=true , address=45.91.116.4:4940 , piece_id=G7JVOOU24MGUUAJST54AWGAHJCQ3NEEDNYJCT7274VL3PMXXYEEA, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.424298210 (320h11m8.975768671s) [noise=true , address=145.79.227.28:4725 , piece_id=WAPK2NYLOGLXI4BOM6LA43UNUYKKVGCD6LA2KKPPIMXROR5AHCEA, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.424920144 (320h11m8.976390615s) [noise=true , address=145.79.227.14:4840 , piece_id=IFKFP5EBBMSNGDII2I2MT5IHK223AP2LD2RZLGMI2UFH7EVB6R3A, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.425382976 (320h11m8.976853437s) [noise=true , address=77.90.31.15:4699 , piece_id=6Q26DE27DV2TCYPSVMG64R36I4NJFZ6XMSX6ISYYJI6XD3E2TKQA, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.425904149 (320h11m8.97737461s) [noise=true , address=145.79.227.22:4509 , piece_id=XBHW2Q4ZMTXHM3DXV2L3H3UJRHW5NYMLO2Z5WMBPSR3QEKWM4YYA, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.426380896 (320h11m8.977851367s) [noise=true , address=145.79.227.13:4802 , piece_id=VQ25RRLCCHN2BIEARHDMGO6JXOLM3RAAYSBGDJRZZAA7LU6XIORA, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.427020925 (320h11m8.978491386s) [noise=true , address=145.79.227.21:4474 , piece_id=PECKNOFRUZZIVY5XJ32WCP3CFBJ7LERMD6QFZZPLAXA3B47VNYBQ, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.428223965 (320h11m8.979694426s) [noise=true , address=145.79.227.8:4331 , piece_id=IXCEZX2MCTDJJWMQY67WUQDLVKUBC6ILSZ66OTDRGZCF43DPPGSQ, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.430197163 (320h11m8.981667624s) [noise=true , address=77.90.31.12:4590 , piece_id=TUKND5MYHBQIH5FKBG6KIFDNAEUAGEX4SZ62D2BJFA77U2KU5UKQ, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.432639544 (320h11m8.984109995s) [noise=true , address=77.90.31.18:4807 , piece_id=QDVLUO3SEXUQJCCMJ4O5VNQK2XBTLXTYKKNZXDFTLXBFRUHMEKEQ, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.435162708 (320h11m8.986633169s) [noise=true , address=145.79.227.15:4874 , piece_id=FE5ZYSKFAKRZZ6RVLVKGZN46PBS5BZFZP5PKVYZNZFBXYGFIKFKA, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.437291813 (320h11m8.988762274s) [noise=true , address=145.79.227.6:4259 , piece_id=JFLTPOWA63S22QGQ6LN4WRFYVV2XQIDRLL7VGWFJRMCPXK5M6MXA, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.439344221 (320h11m8.990814682s) [noise=true , address=77.90.31.19:4841 , piece_id=VKYGMN3H4X2JJUFLNDGZZSCV5HCLQRLR53X5J36FGQ565MKWMNCA, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.440379501 (320h11m8.991849973s) [noise=true , address=145.79.227.20:4437 , piece_id=6ION5L5UGDHAJQTYOAC34BTWDUJOJQXDF4XBRF7SQMRY57BYECOQ, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:12.150193113 (320h11m10.701663574s) [upload=281977]: Next returned nil segment Nov 22 16:15:12.150240513 (320h11m10.701710974s) [upload=281977]: Waiting for error group managing segments... Nov 22 16:15:12.562473597 (320h11m11.113944048s) [noise=true , address=77.90.31.15:4699 , piece_id=6Q26DE27DV2TCYPSVMG64R36I4NJFZ6XMSX6ISYYJI6XD3E2TKQA, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:12.579741378 (320h11m11.131211839s) [noise=true , address=77.90.31.12:4590 , piece_id=TUKND5MYHBQIH5FKBG6KIFDNAEUAGEX4SZ62D2BJFA77U2KU5UKQ, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:12.586731719 (320h11m11.13820218s) [noise=true , address=45.91.116.4:4940 , piece_id=G7JVOOU24MGUUAJST54AWGAHJCQ3NEEDNYJCT7274VL3PMXXYEEA, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:12.598257075 (320h11m11.149727536s) [noise=true , address=77.90.31.12:4589 , piece_id=DMJXC2AGYUS6IJDU57YMRVQ4XITS6C65KQPKPK77BAVNIWEDYW6A, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:12.622577225 (320h11m11.174047686s) [noise=true , address=77.90.31.19:4841 , piece_id=VKYGMN3H4X2JJUFLNDGZZSCV5HCLQRLR53X5J36FGQ565MKWMNCA, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:12.754772838 (320h11m11.306243299s) [noise=true , address=77.90.31.18:4807 , piece_id=QDVLUO3SEXUQJCCMJ4O5VNQK2XBTLXTYKKNZXDFTLXBFRUHMEKEQ, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:19.382583530 (320h11m17.934053981s) [noise=true , address=145.79.227.15:4874 , piece_id=FE5ZYSKFAKRZZ6RVLVKGZN46PBS5BZFZP5PKVYZNZFBXYGFIKFKA, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:21.514760656 (320h11m20.066231117s) [noise=true , address=145.79.227.22:4509 , piece_id=XBHW2Q4ZMTXHM3DXV2L3H3UJRHW5NYMLO2Z5WMBPSR3QEKWM4YYA, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:23.636942539 (320h11m22.188413s) [noise=true , address=145.79.227.6:4259 , piece_id=JFLTPOWA63S22QGQ6LN4WRFYVV2XQIDRLL7VGWFJRMCPXK5M6MXA, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:23.872352010 (320h11m22.423822471s) [noise=true , address=145.79.227.28:4725 , piece_id=WAPK2NYLOGLXI4BOM6LA43UNUYKKVGCD6LA2KKPPIMXROR5AHCEA, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:25.966169160 (320h11m24.517639612s) [noise=true , address=145.79.227.8:4331 , piece_id=IXCEZX2MCTDJJWMQY67WUQDLVKUBC6ILSZ66OTDRGZCF43DPPGSQ, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:26.960865746 (320h11m25.512336207s) [noise=true , address=145.79.227.14:4840 , piece_id=IFKFP5EBBMSNGDII2I2MT5IHK223AP2LD2RZLGMI2UFH7EVB6R3A, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:36.322690505 (320h11m34.874160966s) [noise=true , address=145.79.227.21:4474 , piece_id=PECKNOFRUZZIVY5XJ32WCP3CFBJ7LERMD6QFZZPLAXA3B47VNYBQ, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:36.322718137 (320h11m34.874188599s) [seg_pos={0 0}, upload=281977]: Segment reached optimal threshold of 13 pieces. Nov 22 16:15:36.323447516 (320h11m34.874917977s) [noise=true , address=145.79.227.11:4731 , piece_id=ASYJFIYGHG2VMQE5Y2W6J2MH4HE4TJX5HB36VDPNL6JI6LOBOWMA, seg_pos={0 0}, upload=281977]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9): protocol: rpc: dial tcp 145.79.227.11:4731: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 16:15:36.323459900 (320h11m34.874930371s) [noise=true , address=145.79.227.11:4731 , piece_id=ASYJFIYGHG2VMQE5Y2W6J2MH4HE4TJX5HB36VDPNL6JI6LOBOWMA, seg_pos={0 0}, upload=281977]: Long tail cancellation due to optimal threshold Nov 22 16:15:36.323615366 (320h11m34.875085817s) [noise=true , address=145.79.227.20:4437 , piece_id=6ION5L5UGDHAJQTYOAC34BTWDUJOJQXDF4XBRF7SQMRY57BYECOQ, seg_pos={0 0}, upload=281977]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1G3fe4SHMdNAjAmqLNyFb6qDKL94aAWGYE9qhczKmozKFhKKEV): protocol: write tcp 188.166.225.170:49846->145.79.227.20:4437: use of closed network connection; write tcp 188.166.225.170:49846->145.79.227.20:4437: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49846->145.79.227.20:4437: use of closed network connection Nov 22 16:15:36.323624343 (320h11m34.875094804s) [noise=true , address=145.79.227.20:4437 , piece_id=6ION5L5UGDHAJQTYOAC34BTWDUJOJQXDF4XBRF7SQMRY57BYECOQ, seg_pos={0 0}, upload=281977]: Long tail cancellation due to optimal threshold Nov 22 16:15:36.323867195 (320h11m34.875337677s) [noise=true , address=145.79.227.13:4802 , piece_id=VQ25RRLCCHN2BIEARHDMGO6JXOLM3RAAYSBGDJRZZAA7LU6XIORA, seg_pos={0 0}, upload=281977]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4): protocol: read tcp 188.166.225.170:59064->145.79.227.13:4802: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 16:15:36.323877184 (320h11m34.875347645s) [noise=true , address=145.79.227.13:4802 , piece_id=VQ25RRLCCHN2BIEARHDMGO6JXOLM3RAAYSBGDJRZZAA7LU6XIORA, seg_pos={0 0}, upload=281977]: Long tail cancellation due to optimal threshold Nov 22 16:15:36.323899958 (320h11m34.875370419s) [seg_pos={0 0}, upload=281977]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 22 16:15:36.324029074 (320h11m34.875499535s) [upload=281977]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 22 16:15:36.324036789 (320h11m34.87550726s) [upload=281977]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 22 16:15:36.620020293 (320h11m35.171490744s) [upload=281977]: Done uploading segments... Nov 22 16:15:36.620042956 (320h11m35.171513417s) [upload=281977]: Upload finished. err: <nil> Duration: 11.44625393s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 15 out of 60 is failed (threshold=6) [..XXX.XX.......X..XX......X.X...X.......X...........X.X....X] duration: 5.00215284s trace: https://jaeger.dev.storj.io/trace/102f7d28bf9f3520 FAIL (2025-11-22 16:38:36 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.002180231s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................X.......................................] duration: 12.308087771s trace: https://jaeger.dev.storj.io/trace/13bcd3d57614c8f8 FAIL (2025-11-22 16:38:25 UTC) orbiter warning: context deadline exceeded Duration: 12.308116537s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..........................XX................................] duration: 10.610801773s trace: https://jaeger.dev.storj.io/trace/45889407e3b21f30 FAIL (2025-11-22 16:37:56 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ad Nov 22 16:15:10.157378989 (320h11m8.70884945s) [upload=281977]: Uploading segments... Nov 22 16:15:10.157518405 (320h11m8.708988866s) [upload=281977]: Got next segment. Inline: false Nov 22 16:15:10.157568901 (320h11m8.709039373s) [upload=281977]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 22 16:15:10.157576005 (320h11m8.709046466s) [upload=281977]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 22 16:15:10.422893065 (320h11m8.974363526s) [seg_pos={0 0}, upload=281977]: Begin upload segment... Nov 22 16:15:10.422954472 (320h11m8.974424933s) [seg_pos={0 0}, upload=281977]: Done begin upload segment. Nov 22 16:15:10.422989378 (320h11m8.974459839s) [noise=true , address=77.90.31.12:4589 , piece_id=DMJXC2AGYUS6IJDU57YMRVQ4XITS6C65KQPKPK77BAVNIWEDYW6A, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.423082535 (320h11m8.974553026s) [noise=true , address=145.79.227.11:4731 , piece_id=ASYJFIYGHG2VMQE5Y2W6J2MH4HE4TJX5HB36VDPNL6JI6LOBOWMA, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.423840959 (320h11m8.97531142s) [noise=true , address=45.91.116.4:4940 , piece_id=G7JVOOU24MGUUAJST54AWGAHJCQ3NEEDNYJCT7274VL3PMXXYEEA, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.424298210 (320h11m8.975768671s) [noise=true , address=145.79.227.28:4725 , piece_id=WAPK2NYLOGLXI4BOM6LA43UNUYKKVGCD6LA2KKPPIMXROR5AHCEA, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.424920144 (320h11m8.976390615s) [noise=true , address=145.79.227.14:4840 , piece_id=IFKFP5EBBMSNGDII2I2MT5IHK223AP2LD2RZLGMI2UFH7EVB6R3A, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.425382976 (320h11m8.976853437s) [noise=true , address=77.90.31.15:4699 , piece_id=6Q26DE27DV2TCYPSVMG64R36I4NJFZ6XMSX6ISYYJI6XD3E2TKQA, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.425904149 (320h11m8.97737461s) [noise=true , address=145.79.227.22:4509 , piece_id=XBHW2Q4ZMTXHM3DXV2L3H3UJRHW5NYMLO2Z5WMBPSR3QEKWM4YYA, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.426380896 (320h11m8.977851367s) [noise=true , address=145.79.227.13:4802 , piece_id=VQ25RRLCCHN2BIEARHDMGO6JXOLM3RAAYSBGDJRZZAA7LU6XIORA, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.427020925 (320h11m8.978491386s) [noise=true , address=145.79.227.21:4474 , piece_id=PECKNOFRUZZIVY5XJ32WCP3CFBJ7LERMD6QFZZPLAXA3B47VNYBQ, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.428223965 (320h11m8.979694426s) [noise=true , address=145.79.227.8:4331 , piece_id=IXCEZX2MCTDJJWMQY67WUQDLVKUBC6ILSZ66OTDRGZCF43DPPGSQ, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.430197163 (320h11m8.981667624s) [noise=true , address=77.90.31.12:4590 , piece_id=TUKND5MYHBQIH5FKBG6KIFDNAEUAGEX4SZ62D2BJFA77U2KU5UKQ, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.432639544 (320h11m8.984109995s) [noise=true , address=77.90.31.18:4807 , piece_id=QDVLUO3SEXUQJCCMJ4O5VNQK2XBTLXTYKKNZXDFTLXBFRUHMEKEQ, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.435162708 (320h11m8.986633169s) [noise=true , address=145.79.227.15:4874 , piece_id=FE5ZYSKFAKRZZ6RVLVKGZN46PBS5BZFZP5PKVYZNZFBXYGFIKFKA, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.437291813 (320h11m8.988762274s) [noise=true , address=145.79.227.6:4259 , piece_id=JFLTPOWA63S22QGQ6LN4WRFYVV2XQIDRLL7VGWFJRMCPXK5M6MXA, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.439344221 (320h11m8.990814682s) [noise=true , address=77.90.31.19:4841 , piece_id=VKYGMN3H4X2JJUFLNDGZZSCV5HCLQRLR53X5J36FGQ565MKWMNCA, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.440379501 (320h11m8.991849973s) [noise=true , address=145.79.227.20:4437 , piece_id=6ION5L5UGDHAJQTYOAC34BTWDUJOJQXDF4XBRF7SQMRY57BYECOQ, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:12.150193113 (320h11m10.701663574s) [upload=281977]: Next returned nil segment Nov 22 16:15:12.150240513 (320h11m10.701710974s) [upload=281977]: Waiting for error group managing segments... Nov 22 16:15:12.562473597 (320h11m11.113944048s) [noise=true , address=77.90.31.15:4699 , piece_id=6Q26DE27DV2TCYPSVMG64R36I4NJFZ6XMSX6ISYYJI6XD3E2TKQA, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:12.579741378 (320h11m11.131211839s) [noise=true , address=77.90.31.12:4590 , piece_id=TUKND5MYHBQIH5FKBG6KIFDNAEUAGEX4SZ62D2BJFA77U2KU5UKQ, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:12.586731719 (320h11m11.13820218s) [noise=true , address=45.91.116.4:4940 , piece_id=G7JVOOU24MGUUAJST54AWGAHJCQ3NEEDNYJCT7274VL3PMXXYEEA, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:12.598257075 (320h11m11.149727536s) [noise=true , address=77.90.31.12:4589 , piece_id=DMJXC2AGYUS6IJDU57YMRVQ4XITS6C65KQPKPK77BAVNIWEDYW6A, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:12.622577225 (320h11m11.174047686s) [noise=true , address=77.90.31.19:4841 , piece_id=VKYGMN3H4X2JJUFLNDGZZSCV5HCLQRLR53X5J36FGQ565MKWMNCA, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:12.754772838 (320h11m11.306243299s) [noise=true , address=77.90.31.18:4807 , piece_id=QDVLUO3SEXUQJCCMJ4O5VNQK2XBTLXTYKKNZXDFTLXBFRUHMEKEQ, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:19.382583530 (320h11m17.934053981s) [noise=true , address=145.79.227.15:4874 , piece_id=FE5ZYSKFAKRZZ6RVLVKGZN46PBS5BZFZP5PKVYZNZFBXYGFIKFKA, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:21.514760656 (320h11m20.066231117s) [noise=true , address=145.79.227.22:4509 , piece_id=XBHW2Q4ZMTXHM3DXV2L3H3UJRHW5NYMLO2Z5WMBPSR3QEKWM4YYA, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:23.636942539 (320h11m22.188413s) [noise=true , address=145.79.227.6:4259 , piece_id=JFLTPOWA63S22QGQ6LN4WRFYVV2XQIDRLL7VGWFJRMCPXK5M6MXA, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:23.872352010 (320h11m22.423822471s) [noise=true , address=145.79.227.28:4725 , piece_id=WAPK2NYLOGLXI4BOM6LA43UNUYKKVGCD6LA2KKPPIMXROR5AHCEA, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:25.966169160 (320h11m24.517639612s) [noise=true , address=145.79.227.8:4331 , piece_id=IXCEZX2MCTDJJWMQY67WUQDLVKUBC6ILSZ66OTDRGZCF43DPPGSQ, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:26.960865746 (320h11m25.512336207s) [noise=true , address=145.79.227.14:4840 , piece_id=IFKFP5EBBMSNGDII2I2MT5IHK223AP2LD2RZLGMI2UFH7EVB6R3A, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:36.322690505 (320h11m34.874160966s) [noise=true , address=145.79.227.21:4474 , piece_id=PECKNOFRUZZIVY5XJ32WCP3CFBJ7LERMD6QFZZPLAXA3B47VNYBQ, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:36.322718137 (320h11m34.874188599s) [seg_pos={0 0}, upload=281977]: Segment reached optimal threshold of 13 pieces. Nov 22 16:15:36.323447516 (320h11m34.874917977s) [noise=true , address=145.79.227.11:4731 , piece_id=ASYJFIYGHG2VMQE5Y2W6J2MH4HE4TJX5HB36VDPNL6JI6LOBOWMA, seg_pos={0 0}, upload=281977]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9): protocol: rpc: dial tcp 145.79.227.11:4731: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 16:15:36.323459900 (320h11m34.874930371s) [noise=true , address=145.79.227.11:4731 , piece_id=ASYJFIYGHG2VMQE5Y2W6J2MH4HE4TJX5HB36VDPNL6JI6LOBOWMA, seg_pos={0 0}, upload=281977]: Long tail cancellation due to optimal threshold Nov 22 16:15:36.323615366 (320h11m34.875085817s) [noise=true , address=145.79.227.20:4437 , piece_id=6ION5L5UGDHAJQTYOAC34BTWDUJOJQXDF4XBRF7SQMRY57BYECOQ, seg_pos={0 0}, upload=281977]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1G3fe4SHMdNAjAmqLNyFb6qDKL94aAWGYE9qhczKmozKFhKKEV): protocol: write tcp 188.166.225.170:49846->145.79.227.20:4437: use of closed network connection; write tcp 188.166.225.170:49846->145.79.227.20:4437: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49846->145.79.227.20:4437: use of closed network connection Nov 22 16:15:36.323624343 (320h11m34.875094804s) [noise=true , address=145.79.227.20:4437 , piece_id=6ION5L5UGDHAJQTYOAC34BTWDUJOJQXDF4XBRF7SQMRY57BYECOQ, seg_pos={0 0}, upload=281977]: Long tail cancellation due to optimal threshold Nov 22 16:15:36.323867195 (320h11m34.875337677s) [noise=true , address=145.79.227.13:4802 , piece_id=VQ25RRLCCHN2BIEARHDMGO6JXOLM3RAAYSBGDJRZZAA7LU6XIORA, seg_pos={0 0}, upload=281977]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4): protocol: read tcp 188.166.225.170:59064->145.79.227.13:4802: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 16:15:36.323877184 (320h11m34.875347645s) [noise=true , address=145.79.227.13:4802 , piece_id=VQ25RRLCCHN2BIEARHDMGO6JXOLM3RAAYSBGDJRZZAA7LU6XIORA, seg_pos={0 0}, upload=281977]: Long tail cancellation due to optimal threshold Nov 22 16:15:36.323899958 (320h11m34.875370419s) [seg_pos={0 0}, upload=281977]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 22 16:15:36.324029074 (320h11m34.875499535s) [upload=281977]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 22 16:15:36.324036789 (320h11m34.87550726s) [upload=281977]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 22 16:15:36.620020293 (320h11m35.171490744s) [upload=281977]: Done uploading segments... Nov 22 16:15:36.620042956 (320h11m35.171513417s) [upload=281977]: Upload finished. err: <nil> Duration: 10.610829566s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 14 out of 60 is failed (threshold=6) [...XXX.XX.......X..XX......X.X...X.......X...........X.X....] duration: 3.538424255s trace: https://jaeger.dev.storj.io/trace/5c1ee2fb57171013 FAIL (2025-11-22 16:37:55 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.538451457s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 14 out of 60 is failed (threshold=6) [....XXX.XX.......X..XX......X.X...X.......X...........X.X...] duration: 3.583516719s trace: https://jaeger.dev.storj.io/trace/5c1ee2fb57171013 FAIL (2025-11-22 16:37:29 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.583546336s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................X......................................] duration: 7.014562225s trace: https://jaeger.dev.storj.io/trace/13bcd3d57614c8f8 FAIL (2025-11-22 16:37:08 UTC) orbiter warning: context deadline exceeded Duration: 7.014635564s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 14 out of 60 is failed (threshold=6) [.....XXX.XX.......X..XX......X.X...X.......X...........X.X..] duration: 3.663230318s trace: https://jaeger.dev.storj.io/trace/5c1ee2fb57171013 FAIL (2025-11-22 16:37:04 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.663261858s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...........................XX...............................] duration: 14.849795285s trace: https://jaeger.dev.storj.io/trace/45889407e3b21f30 FAIL (2025-11-22 16:36:52 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ad Nov 22 16:15:10.157378989 (320h11m8.70884945s) [upload=281977]: Uploading segments... Nov 22 16:15:10.157518405 (320h11m8.708988866s) [upload=281977]: Got next segment. Inline: false Nov 22 16:15:10.157568901 (320h11m8.709039373s) [upload=281977]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 22 16:15:10.157576005 (320h11m8.709046466s) [upload=281977]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 22 16:15:10.422893065 (320h11m8.974363526s) [seg_pos={0 0}, upload=281977]: Begin upload segment... Nov 22 16:15:10.422954472 (320h11m8.974424933s) [seg_pos={0 0}, upload=281977]: Done begin upload segment. Nov 22 16:15:10.422989378 (320h11m8.974459839s) [noise=true , address=77.90.31.12:4589 , piece_id=DMJXC2AGYUS6IJDU57YMRVQ4XITS6C65KQPKPK77BAVNIWEDYW6A, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.423082535 (320h11m8.974553026s) [noise=true , address=145.79.227.11:4731 , piece_id=ASYJFIYGHG2VMQE5Y2W6J2MH4HE4TJX5HB36VDPNL6JI6LOBOWMA, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.423840959 (320h11m8.97531142s) [noise=true , address=45.91.116.4:4940 , piece_id=G7JVOOU24MGUUAJST54AWGAHJCQ3NEEDNYJCT7274VL3PMXXYEEA, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.424298210 (320h11m8.975768671s) [noise=true , address=145.79.227.28:4725 , piece_id=WAPK2NYLOGLXI4BOM6LA43UNUYKKVGCD6LA2KKPPIMXROR5AHCEA, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.424920144 (320h11m8.976390615s) [noise=true , address=145.79.227.14:4840 , piece_id=IFKFP5EBBMSNGDII2I2MT5IHK223AP2LD2RZLGMI2UFH7EVB6R3A, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.425382976 (320h11m8.976853437s) [noise=true , address=77.90.31.15:4699 , piece_id=6Q26DE27DV2TCYPSVMG64R36I4NJFZ6XMSX6ISYYJI6XD3E2TKQA, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.425904149 (320h11m8.97737461s) [noise=true , address=145.79.227.22:4509 , piece_id=XBHW2Q4ZMTXHM3DXV2L3H3UJRHW5NYMLO2Z5WMBPSR3QEKWM4YYA, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.426380896 (320h11m8.977851367s) [noise=true , address=145.79.227.13:4802 , piece_id=VQ25RRLCCHN2BIEARHDMGO6JXOLM3RAAYSBGDJRZZAA7LU6XIORA, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.427020925 (320h11m8.978491386s) [noise=true , address=145.79.227.21:4474 , piece_id=PECKNOFRUZZIVY5XJ32WCP3CFBJ7LERMD6QFZZPLAXA3B47VNYBQ, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.428223965 (320h11m8.979694426s) [noise=true , address=145.79.227.8:4331 , piece_id=IXCEZX2MCTDJJWMQY67WUQDLVKUBC6ILSZ66OTDRGZCF43DPPGSQ, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.430197163 (320h11m8.981667624s) [noise=true , address=77.90.31.12:4590 , piece_id=TUKND5MYHBQIH5FKBG6KIFDNAEUAGEX4SZ62D2BJFA77U2KU5UKQ, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.432639544 (320h11m8.984109995s) [noise=true , address=77.90.31.18:4807 , piece_id=QDVLUO3SEXUQJCCMJ4O5VNQK2XBTLXTYKKNZXDFTLXBFRUHMEKEQ, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.435162708 (320h11m8.986633169s) [noise=true , address=145.79.227.15:4874 , piece_id=FE5ZYSKFAKRZZ6RVLVKGZN46PBS5BZFZP5PKVYZNZFBXYGFIKFKA, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.437291813 (320h11m8.988762274s) [noise=true , address=145.79.227.6:4259 , piece_id=JFLTPOWA63S22QGQ6LN4WRFYVV2XQIDRLL7VGWFJRMCPXK5M6MXA, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.439344221 (320h11m8.990814682s) [noise=true , address=77.90.31.19:4841 , piece_id=VKYGMN3H4X2JJUFLNDGZZSCV5HCLQRLR53X5J36FGQ565MKWMNCA, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.440379501 (320h11m8.991849973s) [noise=true , address=145.79.227.20:4437 , piece_id=6ION5L5UGDHAJQTYOAC34BTWDUJOJQXDF4XBRF7SQMRY57BYECOQ, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:12.150193113 (320h11m10.701663574s) [upload=281977]: Next returned nil segment Nov 22 16:15:12.150240513 (320h11m10.701710974s) [upload=281977]: Waiting for error group managing segments... Nov 22 16:15:12.562473597 (320h11m11.113944048s) [noise=true , address=77.90.31.15:4699 , piece_id=6Q26DE27DV2TCYPSVMG64R36I4NJFZ6XMSX6ISYYJI6XD3E2TKQA, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:12.579741378 (320h11m11.131211839s) [noise=true , address=77.90.31.12:4590 , piece_id=TUKND5MYHBQIH5FKBG6KIFDNAEUAGEX4SZ62D2BJFA77U2KU5UKQ, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:12.586731719 (320h11m11.13820218s) [noise=true , address=45.91.116.4:4940 , piece_id=G7JVOOU24MGUUAJST54AWGAHJCQ3NEEDNYJCT7274VL3PMXXYEEA, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:12.598257075 (320h11m11.149727536s) [noise=true , address=77.90.31.12:4589 , piece_id=DMJXC2AGYUS6IJDU57YMRVQ4XITS6C65KQPKPK77BAVNIWEDYW6A, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:12.622577225 (320h11m11.174047686s) [noise=true , address=77.90.31.19:4841 , piece_id=VKYGMN3H4X2JJUFLNDGZZSCV5HCLQRLR53X5J36FGQ565MKWMNCA, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:12.754772838 (320h11m11.306243299s) [noise=true , address=77.90.31.18:4807 , piece_id=QDVLUO3SEXUQJCCMJ4O5VNQK2XBTLXTYKKNZXDFTLXBFRUHMEKEQ, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:19.382583530 (320h11m17.934053981s) [noise=true , address=145.79.227.15:4874 , piece_id=FE5ZYSKFAKRZZ6RVLVKGZN46PBS5BZFZP5PKVYZNZFBXYGFIKFKA, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:21.514760656 (320h11m20.066231117s) [noise=true , address=145.79.227.22:4509 , piece_id=XBHW2Q4ZMTXHM3DXV2L3H3UJRHW5NYMLO2Z5WMBPSR3QEKWM4YYA, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:23.636942539 (320h11m22.188413s) [noise=true , address=145.79.227.6:4259 , piece_id=JFLTPOWA63S22QGQ6LN4WRFYVV2XQIDRLL7VGWFJRMCPXK5M6MXA, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:23.872352010 (320h11m22.423822471s) [noise=true , address=145.79.227.28:4725 , piece_id=WAPK2NYLOGLXI4BOM6LA43UNUYKKVGCD6LA2KKPPIMXROR5AHCEA, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:25.966169160 (320h11m24.517639612s) [noise=true , address=145.79.227.8:4331 , piece_id=IXCEZX2MCTDJJWMQY67WUQDLVKUBC6ILSZ66OTDRGZCF43DPPGSQ, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:26.960865746 (320h11m25.512336207s) [noise=true , address=145.79.227.14:4840 , piece_id=IFKFP5EBBMSNGDII2I2MT5IHK223AP2LD2RZLGMI2UFH7EVB6R3A, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:36.322690505 (320h11m34.874160966s) [noise=true , address=145.79.227.21:4474 , piece_id=PECKNOFRUZZIVY5XJ32WCP3CFBJ7LERMD6QFZZPLAXA3B47VNYBQ, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:36.322718137 (320h11m34.874188599s) [seg_pos={0 0}, upload=281977]: Segment reached optimal threshold of 13 pieces. Nov 22 16:15:36.323447516 (320h11m34.874917977s) [noise=true , address=145.79.227.11:4731 , piece_id=ASYJFIYGHG2VMQE5Y2W6J2MH4HE4TJX5HB36VDPNL6JI6LOBOWMA, seg_pos={0 0}, upload=281977]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9): protocol: rpc: dial tcp 145.79.227.11:4731: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 16:15:36.323459900 (320h11m34.874930371s) [noise=true , address=145.79.227.11:4731 , piece_id=ASYJFIYGHG2VMQE5Y2W6J2MH4HE4TJX5HB36VDPNL6JI6LOBOWMA, seg_pos={0 0}, upload=281977]: Long tail cancellation due to optimal threshold Nov 22 16:15:36.323615366 (320h11m34.875085817s) [noise=true , address=145.79.227.20:4437 , piece_id=6ION5L5UGDHAJQTYOAC34BTWDUJOJQXDF4XBRF7SQMRY57BYECOQ, seg_pos={0 0}, upload=281977]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1G3fe4SHMdNAjAmqLNyFb6qDKL94aAWGYE9qhczKmozKFhKKEV): protocol: write tcp 188.166.225.170:49846->145.79.227.20:4437: use of closed network connection; write tcp 188.166.225.170:49846->145.79.227.20:4437: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49846->145.79.227.20:4437: use of closed network connection Nov 22 16:15:36.323624343 (320h11m34.875094804s) [noise=true , address=145.79.227.20:4437 , piece_id=6ION5L5UGDHAJQTYOAC34BTWDUJOJQXDF4XBRF7SQMRY57BYECOQ, seg_pos={0 0}, upload=281977]: Long tail cancellation due to optimal threshold Nov 22 16:15:36.323867195 (320h11m34.875337677s) [noise=true , address=145.79.227.13:4802 , piece_id=VQ25RRLCCHN2BIEARHDMGO6JXOLM3RAAYSBGDJRZZAA7LU6XIORA, seg_pos={0 0}, upload=281977]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4): protocol: read tcp 188.166.225.170:59064->145.79.227.13:4802: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 16:15:36.323877184 (320h11m34.875347645s) [noise=true , address=145.79.227.13:4802 , piece_id=VQ25RRLCCHN2BIEARHDMGO6JXOLM3RAAYSBGDJRZZAA7LU6XIORA, seg_pos={0 0}, upload=281977]: Long tail cancellation due to optimal threshold Nov 22 16:15:36.323899958 (320h11m34.875370419s) [seg_pos={0 0}, upload=281977]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 22 16:15:36.324029074 (320h11m34.875499535s) [upload=281977]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 22 16:15:36.324036789 (320h11m34.87550726s) [upload=281977]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 22 16:15:36.620020293 (320h11m35.171490744s) [upload=281977]: Done uploading segments... Nov 22 16:15:36.620042956 (320h11m35.171513417s) [upload=281977]: Upload finished. err: <nil> Duration: 14.849816285s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................X.....................................] duration: 13.43848926s trace: https://jaeger.dev.storj.io/trace/13bcd3d57614c8f8 FAIL (2025-11-22 16:36:19 UTC) orbiter warning: context deadline exceeded Duration: 13.438521081s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 14 out of 60 is failed (threshold=6) [......XXX.XX.......X..XX......X.X...X.......X...........X.X.] duration: 3.63612801s trace: https://jaeger.dev.storj.io/trace/5c1ee2fb57171013 FAIL (2025-11-22 16:36:11 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.636162886s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [............................XX..............................] duration: 13.950593256s trace: https://jaeger.dev.storj.io/trace/45889407e3b21f30 FAIL (2025-11-22 16:36:06 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ad Nov 22 16:15:10.157378989 (320h11m8.70884945s) [upload=281977]: Uploading segments... Nov 22 16:15:10.157518405 (320h11m8.708988866s) [upload=281977]: Got next segment. Inline: false Nov 22 16:15:10.157568901 (320h11m8.709039373s) [upload=281977]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 22 16:15:10.157576005 (320h11m8.709046466s) [upload=281977]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 22 16:15:10.422893065 (320h11m8.974363526s) [seg_pos={0 0}, upload=281977]: Begin upload segment... Nov 22 16:15:10.422954472 (320h11m8.974424933s) [seg_pos={0 0}, upload=281977]: Done begin upload segment. Nov 22 16:15:10.422989378 (320h11m8.974459839s) [noise=true , address=77.90.31.12:4589 , piece_id=DMJXC2AGYUS6IJDU57YMRVQ4XITS6C65KQPKPK77BAVNIWEDYW6A, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.423082535 (320h11m8.974553026s) [noise=true , address=145.79.227.11:4731 , piece_id=ASYJFIYGHG2VMQE5Y2W6J2MH4HE4TJX5HB36VDPNL6JI6LOBOWMA, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.423840959 (320h11m8.97531142s) [noise=true , address=45.91.116.4:4940 , piece_id=G7JVOOU24MGUUAJST54AWGAHJCQ3NEEDNYJCT7274VL3PMXXYEEA, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.424298210 (320h11m8.975768671s) [noise=true , address=145.79.227.28:4725 , piece_id=WAPK2NYLOGLXI4BOM6LA43UNUYKKVGCD6LA2KKPPIMXROR5AHCEA, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.424920144 (320h11m8.976390615s) [noise=true , address=145.79.227.14:4840 , piece_id=IFKFP5EBBMSNGDII2I2MT5IHK223AP2LD2RZLGMI2UFH7EVB6R3A, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.425382976 (320h11m8.976853437s) [noise=true , address=77.90.31.15:4699 , piece_id=6Q26DE27DV2TCYPSVMG64R36I4NJFZ6XMSX6ISYYJI6XD3E2TKQA, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.425904149 (320h11m8.97737461s) [noise=true , address=145.79.227.22:4509 , piece_id=XBHW2Q4ZMTXHM3DXV2L3H3UJRHW5NYMLO2Z5WMBPSR3QEKWM4YYA, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.426380896 (320h11m8.977851367s) [noise=true , address=145.79.227.13:4802 , piece_id=VQ25RRLCCHN2BIEARHDMGO6JXOLM3RAAYSBGDJRZZAA7LU6XIORA, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.427020925 (320h11m8.978491386s) [noise=true , address=145.79.227.21:4474 , piece_id=PECKNOFRUZZIVY5XJ32WCP3CFBJ7LERMD6QFZZPLAXA3B47VNYBQ, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.428223965 (320h11m8.979694426s) [noise=true , address=145.79.227.8:4331 , piece_id=IXCEZX2MCTDJJWMQY67WUQDLVKUBC6ILSZ66OTDRGZCF43DPPGSQ, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.430197163 (320h11m8.981667624s) [noise=true , address=77.90.31.12:4590 , piece_id=TUKND5MYHBQIH5FKBG6KIFDNAEUAGEX4SZ62D2BJFA77U2KU5UKQ, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.432639544 (320h11m8.984109995s) [noise=true , address=77.90.31.18:4807 , piece_id=QDVLUO3SEXUQJCCMJ4O5VNQK2XBTLXTYKKNZXDFTLXBFRUHMEKEQ, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.435162708 (320h11m8.986633169s) [noise=true , address=145.79.227.15:4874 , piece_id=FE5ZYSKFAKRZZ6RVLVKGZN46PBS5BZFZP5PKVYZNZFBXYGFIKFKA, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.437291813 (320h11m8.988762274s) [noise=true , address=145.79.227.6:4259 , piece_id=JFLTPOWA63S22QGQ6LN4WRFYVV2XQIDRLL7VGWFJRMCPXK5M6MXA, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.439344221 (320h11m8.990814682s) [noise=true , address=77.90.31.19:4841 , piece_id=VKYGMN3H4X2JJUFLNDGZZSCV5HCLQRLR53X5J36FGQ565MKWMNCA, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.440379501 (320h11m8.991849973s) [noise=true , address=145.79.227.20:4437 , piece_id=6ION5L5UGDHAJQTYOAC34BTWDUJOJQXDF4XBRF7SQMRY57BYECOQ, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:12.150193113 (320h11m10.701663574s) [upload=281977]: Next returned nil segment Nov 22 16:15:12.150240513 (320h11m10.701710974s) [upload=281977]: Waiting for error group managing segments... Nov 22 16:15:12.562473597 (320h11m11.113944048s) [noise=true , address=77.90.31.15:4699 , piece_id=6Q26DE27DV2TCYPSVMG64R36I4NJFZ6XMSX6ISYYJI6XD3E2TKQA, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:12.579741378 (320h11m11.131211839s) [noise=true , address=77.90.31.12:4590 , piece_id=TUKND5MYHBQIH5FKBG6KIFDNAEUAGEX4SZ62D2BJFA77U2KU5UKQ, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:12.586731719 (320h11m11.13820218s) [noise=true , address=45.91.116.4:4940 , piece_id=G7JVOOU24MGUUAJST54AWGAHJCQ3NEEDNYJCT7274VL3PMXXYEEA, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:12.598257075 (320h11m11.149727536s) [noise=true , address=77.90.31.12:4589 , piece_id=DMJXC2AGYUS6IJDU57YMRVQ4XITS6C65KQPKPK77BAVNIWEDYW6A, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:12.622577225 (320h11m11.174047686s) [noise=true , address=77.90.31.19:4841 , piece_id=VKYGMN3H4X2JJUFLNDGZZSCV5HCLQRLR53X5J36FGQ565MKWMNCA, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:12.754772838 (320h11m11.306243299s) [noise=true , address=77.90.31.18:4807 , piece_id=QDVLUO3SEXUQJCCMJ4O5VNQK2XBTLXTYKKNZXDFTLXBFRUHMEKEQ, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:19.382583530 (320h11m17.934053981s) [noise=true , address=145.79.227.15:4874 , piece_id=FE5ZYSKFAKRZZ6RVLVKGZN46PBS5BZFZP5PKVYZNZFBXYGFIKFKA, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:21.514760656 (320h11m20.066231117s) [noise=true , address=145.79.227.22:4509 , piece_id=XBHW2Q4ZMTXHM3DXV2L3H3UJRHW5NYMLO2Z5WMBPSR3QEKWM4YYA, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:23.636942539 (320h11m22.188413s) [noise=true , address=145.79.227.6:4259 , piece_id=JFLTPOWA63S22QGQ6LN4WRFYVV2XQIDRLL7VGWFJRMCPXK5M6MXA, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:23.872352010 (320h11m22.423822471s) [noise=true , address=145.79.227.28:4725 , piece_id=WAPK2NYLOGLXI4BOM6LA43UNUYKKVGCD6LA2KKPPIMXROR5AHCEA, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:25.966169160 (320h11m24.517639612s) [noise=true , address=145.79.227.8:4331 , piece_id=IXCEZX2MCTDJJWMQY67WUQDLVKUBC6ILSZ66OTDRGZCF43DPPGSQ, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:26.960865746 (320h11m25.512336207s) [noise=true , address=145.79.227.14:4840 , piece_id=IFKFP5EBBMSNGDII2I2MT5IHK223AP2LD2RZLGMI2UFH7EVB6R3A, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:36.322690505 (320h11m34.874160966s) [noise=true , address=145.79.227.21:4474 , piece_id=PECKNOFRUZZIVY5XJ32WCP3CFBJ7LERMD6QFZZPLAXA3B47VNYBQ, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:36.322718137 (320h11m34.874188599s) [seg_pos={0 0}, upload=281977]: Segment reached optimal threshold of 13 pieces. Nov 22 16:15:36.323447516 (320h11m34.874917977s) [noise=true , address=145.79.227.11:4731 , piece_id=ASYJFIYGHG2VMQE5Y2W6J2MH4HE4TJX5HB36VDPNL6JI6LOBOWMA, seg_pos={0 0}, upload=281977]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9): protocol: rpc: dial tcp 145.79.227.11:4731: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 16:15:36.323459900 (320h11m34.874930371s) [noise=true , address=145.79.227.11:4731 , piece_id=ASYJFIYGHG2VMQE5Y2W6J2MH4HE4TJX5HB36VDPNL6JI6LOBOWMA, seg_pos={0 0}, upload=281977]: Long tail cancellation due to optimal threshold Nov 22 16:15:36.323615366 (320h11m34.875085817s) [noise=true , address=145.79.227.20:4437 , piece_id=6ION5L5UGDHAJQTYOAC34BTWDUJOJQXDF4XBRF7SQMRY57BYECOQ, seg_pos={0 0}, upload=281977]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1G3fe4SHMdNAjAmqLNyFb6qDKL94aAWGYE9qhczKmozKFhKKEV): protocol: write tcp 188.166.225.170:49846->145.79.227.20:4437: use of closed network connection; write tcp 188.166.225.170:49846->145.79.227.20:4437: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49846->145.79.227.20:4437: use of closed network connection Nov 22 16:15:36.323624343 (320h11m34.875094804s) [noise=true , address=145.79.227.20:4437 , piece_id=6ION5L5UGDHAJQTYOAC34BTWDUJOJQXDF4XBRF7SQMRY57BYECOQ, seg_pos={0 0}, upload=281977]: Long tail cancellation due to optimal threshold Nov 22 16:15:36.323867195 (320h11m34.875337677s) [noise=true , address=145.79.227.13:4802 , piece_id=VQ25RRLCCHN2BIEARHDMGO6JXOLM3RAAYSBGDJRZZAA7LU6XIORA, seg_pos={0 0}, upload=281977]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4): protocol: read tcp 188.166.225.170:59064->145.79.227.13:4802: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 16:15:36.323877184 (320h11m34.875347645s) [noise=true , address=145.79.227.13:4802 , piece_id=VQ25RRLCCHN2BIEARHDMGO6JXOLM3RAAYSBGDJRZZAA7LU6XIORA, seg_pos={0 0}, upload=281977]: Long tail cancellation due to optimal threshold Nov 22 16:15:36.323899958 (320h11m34.875370419s) [seg_pos={0 0}, upload=281977]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 22 16:15:36.324029074 (320h11m34.875499535s) [upload=281977]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 22 16:15:36.324036789 (320h11m34.87550726s) [upload=281977]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 22 16:15:36.620020293 (320h11m35.171490744s) [upload=281977]: Done uploading segments... Nov 22 16:15:36.620042956 (320h11m35.171513417s) [upload=281977]: Upload finished. err: <nil> Duration: 13.950622262s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................X....................................] duration: 6.997407643s trace: https://jaeger.dev.storj.io/trace/13bcd3d57614c8f8 FAIL (2025-11-22 16:35:40 UTC) orbiter warning: context deadline exceeded Duration: 6.997461214s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 14 out of 60 is failed (threshold=6) [.......XXX.XX.......X..XX......X.X...X.......X...........X.X] duration: 5.001895109s trace: https://jaeger.dev.storj.io/trace/5c1ee2fb57171013 FAIL (2025-11-22 16:35:28 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001925926s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.............................XX.............................] duration: 14.518928154s trace: https://jaeger.dev.storj.io/trace/45889407e3b21f30 FAIL (2025-11-22 16:35:22 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ad Nov 22 16:15:10.157378989 (320h11m8.70884945s) [upload=281977]: Uploading segments... Nov 22 16:15:10.157518405 (320h11m8.708988866s) [upload=281977]: Got next segment. Inline: false Nov 22 16:15:10.157568901 (320h11m8.709039373s) [upload=281977]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 22 16:15:10.157576005 (320h11m8.709046466s) [upload=281977]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 22 16:15:10.422893065 (320h11m8.974363526s) [seg_pos={0 0}, upload=281977]: Begin upload segment... Nov 22 16:15:10.422954472 (320h11m8.974424933s) [seg_pos={0 0}, upload=281977]: Done begin upload segment. Nov 22 16:15:10.422989378 (320h11m8.974459839s) [noise=true , address=77.90.31.12:4589 , piece_id=DMJXC2AGYUS6IJDU57YMRVQ4XITS6C65KQPKPK77BAVNIWEDYW6A, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.423082535 (320h11m8.974553026s) [noise=true , address=145.79.227.11:4731 , piece_id=ASYJFIYGHG2VMQE5Y2W6J2MH4HE4TJX5HB36VDPNL6JI6LOBOWMA, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.423840959 (320h11m8.97531142s) [noise=true , address=45.91.116.4:4940 , piece_id=G7JVOOU24MGUUAJST54AWGAHJCQ3NEEDNYJCT7274VL3PMXXYEEA, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.424298210 (320h11m8.975768671s) [noise=true , address=145.79.227.28:4725 , piece_id=WAPK2NYLOGLXI4BOM6LA43UNUYKKVGCD6LA2KKPPIMXROR5AHCEA, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.424920144 (320h11m8.976390615s) [noise=true , address=145.79.227.14:4840 , piece_id=IFKFP5EBBMSNGDII2I2MT5IHK223AP2LD2RZLGMI2UFH7EVB6R3A, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.425382976 (320h11m8.976853437s) [noise=true , address=77.90.31.15:4699 , piece_id=6Q26DE27DV2TCYPSVMG64R36I4NJFZ6XMSX6ISYYJI6XD3E2TKQA, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.425904149 (320h11m8.97737461s) [noise=true , address=145.79.227.22:4509 , piece_id=XBHW2Q4ZMTXHM3DXV2L3H3UJRHW5NYMLO2Z5WMBPSR3QEKWM4YYA, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.426380896 (320h11m8.977851367s) [noise=true , address=145.79.227.13:4802 , piece_id=VQ25RRLCCHN2BIEARHDMGO6JXOLM3RAAYSBGDJRZZAA7LU6XIORA, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.427020925 (320h11m8.978491386s) [noise=true , address=145.79.227.21:4474 , piece_id=PECKNOFRUZZIVY5XJ32WCP3CFBJ7LERMD6QFZZPLAXA3B47VNYBQ, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.428223965 (320h11m8.979694426s) [noise=true , address=145.79.227.8:4331 , piece_id=IXCEZX2MCTDJJWMQY67WUQDLVKUBC6ILSZ66OTDRGZCF43DPPGSQ, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.430197163 (320h11m8.981667624s) [noise=true , address=77.90.31.12:4590 , piece_id=TUKND5MYHBQIH5FKBG6KIFDNAEUAGEX4SZ62D2BJFA77U2KU5UKQ, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.432639544 (320h11m8.984109995s) [noise=true , address=77.90.31.18:4807 , piece_id=QDVLUO3SEXUQJCCMJ4O5VNQK2XBTLXTYKKNZXDFTLXBFRUHMEKEQ, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.435162708 (320h11m8.986633169s) [noise=true , address=145.79.227.15:4874 , piece_id=FE5ZYSKFAKRZZ6RVLVKGZN46PBS5BZFZP5PKVYZNZFBXYGFIKFKA, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.437291813 (320h11m8.988762274s) [noise=true , address=145.79.227.6:4259 , piece_id=JFLTPOWA63S22QGQ6LN4WRFYVV2XQIDRLL7VGWFJRMCPXK5M6MXA, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.439344221 (320h11m8.990814682s) [noise=true , address=77.90.31.19:4841 , piece_id=VKYGMN3H4X2JJUFLNDGZZSCV5HCLQRLR53X5J36FGQ565MKWMNCA, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.440379501 (320h11m8.991849973s) [noise=true , address=145.79.227.20:4437 , piece_id=6ION5L5UGDHAJQTYOAC34BTWDUJOJQXDF4XBRF7SQMRY57BYECOQ, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:12.150193113 (320h11m10.701663574s) [upload=281977]: Next returned nil segment Nov 22 16:15:12.150240513 (320h11m10.701710974s) [upload=281977]: Waiting for error group managing segments... Nov 22 16:15:12.562473597 (320h11m11.113944048s) [noise=true , address=77.90.31.15:4699 , piece_id=6Q26DE27DV2TCYPSVMG64R36I4NJFZ6XMSX6ISYYJI6XD3E2TKQA, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:12.579741378 (320h11m11.131211839s) [noise=true , address=77.90.31.12:4590 , piece_id=TUKND5MYHBQIH5FKBG6KIFDNAEUAGEX4SZ62D2BJFA77U2KU5UKQ, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:12.586731719 (320h11m11.13820218s) [noise=true , address=45.91.116.4:4940 , piece_id=G7JVOOU24MGUUAJST54AWGAHJCQ3NEEDNYJCT7274VL3PMXXYEEA, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:12.598257075 (320h11m11.149727536s) [noise=true , address=77.90.31.12:4589 , piece_id=DMJXC2AGYUS6IJDU57YMRVQ4XITS6C65KQPKPK77BAVNIWEDYW6A, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:12.622577225 (320h11m11.174047686s) [noise=true , address=77.90.31.19:4841 , piece_id=VKYGMN3H4X2JJUFLNDGZZSCV5HCLQRLR53X5J36FGQ565MKWMNCA, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:12.754772838 (320h11m11.306243299s) [noise=true , address=77.90.31.18:4807 , piece_id=QDVLUO3SEXUQJCCMJ4O5VNQK2XBTLXTYKKNZXDFTLXBFRUHMEKEQ, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:19.382583530 (320h11m17.934053981s) [noise=true , address=145.79.227.15:4874 , piece_id=FE5ZYSKFAKRZZ6RVLVKGZN46PBS5BZFZP5PKVYZNZFBXYGFIKFKA, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:21.514760656 (320h11m20.066231117s) [noise=true , address=145.79.227.22:4509 , piece_id=XBHW2Q4ZMTXHM3DXV2L3H3UJRHW5NYMLO2Z5WMBPSR3QEKWM4YYA, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:23.636942539 (320h11m22.188413s) [noise=true , address=145.79.227.6:4259 , piece_id=JFLTPOWA63S22QGQ6LN4WRFYVV2XQIDRLL7VGWFJRMCPXK5M6MXA, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:23.872352010 (320h11m22.423822471s) [noise=true , address=145.79.227.28:4725 , piece_id=WAPK2NYLOGLXI4BOM6LA43UNUYKKVGCD6LA2KKPPIMXROR5AHCEA, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:25.966169160 (320h11m24.517639612s) [noise=true , address=145.79.227.8:4331 , piece_id=IXCEZX2MCTDJJWMQY67WUQDLVKUBC6ILSZ66OTDRGZCF43DPPGSQ, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:26.960865746 (320h11m25.512336207s) [noise=true , address=145.79.227.14:4840 , piece_id=IFKFP5EBBMSNGDII2I2MT5IHK223AP2LD2RZLGMI2UFH7EVB6R3A, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:36.322690505 (320h11m34.874160966s) [noise=true , address=145.79.227.21:4474 , piece_id=PECKNOFRUZZIVY5XJ32WCP3CFBJ7LERMD6QFZZPLAXA3B47VNYBQ, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:36.322718137 (320h11m34.874188599s) [seg_pos={0 0}, upload=281977]: Segment reached optimal threshold of 13 pieces. Nov 22 16:15:36.323447516 (320h11m34.874917977s) [noise=true , address=145.79.227.11:4731 , piece_id=ASYJFIYGHG2VMQE5Y2W6J2MH4HE4TJX5HB36VDPNL6JI6LOBOWMA, seg_pos={0 0}, upload=281977]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9): protocol: rpc: dial tcp 145.79.227.11:4731: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 16:15:36.323459900 (320h11m34.874930371s) [noise=true , address=145.79.227.11:4731 , piece_id=ASYJFIYGHG2VMQE5Y2W6J2MH4HE4TJX5HB36VDPNL6JI6LOBOWMA, seg_pos={0 0}, upload=281977]: Long tail cancellation due to optimal threshold Nov 22 16:15:36.323615366 (320h11m34.875085817s) [noise=true , address=145.79.227.20:4437 , piece_id=6ION5L5UGDHAJQTYOAC34BTWDUJOJQXDF4XBRF7SQMRY57BYECOQ, seg_pos={0 0}, upload=281977]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1G3fe4SHMdNAjAmqLNyFb6qDKL94aAWGYE9qhczKmozKFhKKEV): protocol: write tcp 188.166.225.170:49846->145.79.227.20:4437: use of closed network connection; write tcp 188.166.225.170:49846->145.79.227.20:4437: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49846->145.79.227.20:4437: use of closed network connection Nov 22 16:15:36.323624343 (320h11m34.875094804s) [noise=true , address=145.79.227.20:4437 , piece_id=6ION5L5UGDHAJQTYOAC34BTWDUJOJQXDF4XBRF7SQMRY57BYECOQ, seg_pos={0 0}, upload=281977]: Long tail cancellation due to optimal threshold Nov 22 16:15:36.323867195 (320h11m34.875337677s) [noise=true , address=145.79.227.13:4802 , piece_id=VQ25RRLCCHN2BIEARHDMGO6JXOLM3RAAYSBGDJRZZAA7LU6XIORA, seg_pos={0 0}, upload=281977]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4): protocol: read tcp 188.166.225.170:59064->145.79.227.13:4802: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 16:15:36.323877184 (320h11m34.875347645s) [noise=true , address=145.79.227.13:4802 , piece_id=VQ25RRLCCHN2BIEARHDMGO6JXOLM3RAAYSBGDJRZZAA7LU6XIORA, seg_pos={0 0}, upload=281977]: Long tail cancellation due to optimal threshold Nov 22 16:15:36.323899958 (320h11m34.875370419s) [seg_pos={0 0}, upload=281977]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 22 16:15:36.324029074 (320h11m34.875499535s) [upload=281977]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 22 16:15:36.324036789 (320h11m34.87550726s) [upload=281977]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 22 16:15:36.620020293 (320h11m35.171490744s) [upload=281977]: Done uploading segments... Nov 22 16:15:36.620042956 (320h11m35.171513417s) [upload=281977]: Upload finished. err: <nil> Duration: 14.51897863s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................X...................................] duration: 11.454628335s trace: https://jaeger.dev.storj.io/trace/13bcd3d57614c8f8 FAIL (2025-11-22 16:35:05 UTC) orbiter warning: context deadline exceeded Duration: 11.454657861s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..............................XX............................] duration: 15.37131065s trace: https://jaeger.dev.storj.io/trace/45889407e3b21f30 FAIL (2025-11-22 16:34:55 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ad Nov 22 16:15:10.157378989 (320h11m8.70884945s) [upload=281977]: Uploading segments... Nov 22 16:15:10.157518405 (320h11m8.708988866s) [upload=281977]: Got next segment. Inline: false Nov 22 16:15:10.157568901 (320h11m8.709039373s) [upload=281977]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 22 16:15:10.157576005 (320h11m8.709046466s) [upload=281977]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 22 16:15:10.422893065 (320h11m8.974363526s) [seg_pos={0 0}, upload=281977]: Begin upload segment... Nov 22 16:15:10.422954472 (320h11m8.974424933s) [seg_pos={0 0}, upload=281977]: Done begin upload segment. Nov 22 16:15:10.422989378 (320h11m8.974459839s) [noise=true , address=77.90.31.12:4589 , piece_id=DMJXC2AGYUS6IJDU57YMRVQ4XITS6C65KQPKPK77BAVNIWEDYW6A, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.423082535 (320h11m8.974553026s) [noise=true , address=145.79.227.11:4731 , piece_id=ASYJFIYGHG2VMQE5Y2W6J2MH4HE4TJX5HB36VDPNL6JI6LOBOWMA, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.423840959 (320h11m8.97531142s) [noise=true , address=45.91.116.4:4940 , piece_id=G7JVOOU24MGUUAJST54AWGAHJCQ3NEEDNYJCT7274VL3PMXXYEEA, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.424298210 (320h11m8.975768671s) [noise=true , address=145.79.227.28:4725 , piece_id=WAPK2NYLOGLXI4BOM6LA43UNUYKKVGCD6LA2KKPPIMXROR5AHCEA, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.424920144 (320h11m8.976390615s) [noise=true , address=145.79.227.14:4840 , piece_id=IFKFP5EBBMSNGDII2I2MT5IHK223AP2LD2RZLGMI2UFH7EVB6R3A, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.425382976 (320h11m8.976853437s) [noise=true , address=77.90.31.15:4699 , piece_id=6Q26DE27DV2TCYPSVMG64R36I4NJFZ6XMSX6ISYYJI6XD3E2TKQA, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.425904149 (320h11m8.97737461s) [noise=true , address=145.79.227.22:4509 , piece_id=XBHW2Q4ZMTXHM3DXV2L3H3UJRHW5NYMLO2Z5WMBPSR3QEKWM4YYA, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.426380896 (320h11m8.977851367s) [noise=true , address=145.79.227.13:4802 , piece_id=VQ25RRLCCHN2BIEARHDMGO6JXOLM3RAAYSBGDJRZZAA7LU6XIORA, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.427020925 (320h11m8.978491386s) [noise=true , address=145.79.227.21:4474 , piece_id=PECKNOFRUZZIVY5XJ32WCP3CFBJ7LERMD6QFZZPLAXA3B47VNYBQ, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.428223965 (320h11m8.979694426s) [noise=true , address=145.79.227.8:4331 , piece_id=IXCEZX2MCTDJJWMQY67WUQDLVKUBC6ILSZ66OTDRGZCF43DPPGSQ, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.430197163 (320h11m8.981667624s) [noise=true , address=77.90.31.12:4590 , piece_id=TUKND5MYHBQIH5FKBG6KIFDNAEUAGEX4SZ62D2BJFA77U2KU5UKQ, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.432639544 (320h11m8.984109995s) [noise=true , address=77.90.31.18:4807 , piece_id=QDVLUO3SEXUQJCCMJ4O5VNQK2XBTLXTYKKNZXDFTLXBFRUHMEKEQ, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.435162708 (320h11m8.986633169s) [noise=true , address=145.79.227.15:4874 , piece_id=FE5ZYSKFAKRZZ6RVLVKGZN46PBS5BZFZP5PKVYZNZFBXYGFIKFKA, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.437291813 (320h11m8.988762274s) [noise=true , address=145.79.227.6:4259 , piece_id=JFLTPOWA63S22QGQ6LN4WRFYVV2XQIDRLL7VGWFJRMCPXK5M6MXA, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.439344221 (320h11m8.990814682s) [noise=true , address=77.90.31.19:4841 , piece_id=VKYGMN3H4X2JJUFLNDGZZSCV5HCLQRLR53X5J36FGQ565MKWMNCA, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.440379501 (320h11m8.991849973s) [noise=true , address=145.79.227.20:4437 , piece_id=6ION5L5UGDHAJQTYOAC34BTWDUJOJQXDF4XBRF7SQMRY57BYECOQ, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:12.150193113 (320h11m10.701663574s) [upload=281977]: Next returned nil segment Nov 22 16:15:12.150240513 (320h11m10.701710974s) [upload=281977]: Waiting for error group managing segments... Nov 22 16:15:12.562473597 (320h11m11.113944048s) [noise=true , address=77.90.31.15:4699 , piece_id=6Q26DE27DV2TCYPSVMG64R36I4NJFZ6XMSX6ISYYJI6XD3E2TKQA, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:12.579741378 (320h11m11.131211839s) [noise=true , address=77.90.31.12:4590 , piece_id=TUKND5MYHBQIH5FKBG6KIFDNAEUAGEX4SZ62D2BJFA77U2KU5UKQ, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:12.586731719 (320h11m11.13820218s) [noise=true , address=45.91.116.4:4940 , piece_id=G7JVOOU24MGUUAJST54AWGAHJCQ3NEEDNYJCT7274VL3PMXXYEEA, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:12.598257075 (320h11m11.149727536s) [noise=true , address=77.90.31.12:4589 , piece_id=DMJXC2AGYUS6IJDU57YMRVQ4XITS6C65KQPKPK77BAVNIWEDYW6A, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:12.622577225 (320h11m11.174047686s) [noise=true , address=77.90.31.19:4841 , piece_id=VKYGMN3H4X2JJUFLNDGZZSCV5HCLQRLR53X5J36FGQ565MKWMNCA, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:12.754772838 (320h11m11.306243299s) [noise=true , address=77.90.31.18:4807 , piece_id=QDVLUO3SEXUQJCCMJ4O5VNQK2XBTLXTYKKNZXDFTLXBFRUHMEKEQ, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:19.382583530 (320h11m17.934053981s) [noise=true , address=145.79.227.15:4874 , piece_id=FE5ZYSKFAKRZZ6RVLVKGZN46PBS5BZFZP5PKVYZNZFBXYGFIKFKA, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:21.514760656 (320h11m20.066231117s) [noise=true , address=145.79.227.22:4509 , piece_id=XBHW2Q4ZMTXHM3DXV2L3H3UJRHW5NYMLO2Z5WMBPSR3QEKWM4YYA, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:23.636942539 (320h11m22.188413s) [noise=true , address=145.79.227.6:4259 , piece_id=JFLTPOWA63S22QGQ6LN4WRFYVV2XQIDRLL7VGWFJRMCPXK5M6MXA, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:23.872352010 (320h11m22.423822471s) [noise=true , address=145.79.227.28:4725 , piece_id=WAPK2NYLOGLXI4BOM6LA43UNUYKKVGCD6LA2KKPPIMXROR5AHCEA, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:25.966169160 (320h11m24.517639612s) [noise=true , address=145.79.227.8:4331 , piece_id=IXCEZX2MCTDJJWMQY67WUQDLVKUBC6ILSZ66OTDRGZCF43DPPGSQ, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:26.960865746 (320h11m25.512336207s) [noise=true , address=145.79.227.14:4840 , piece_id=IFKFP5EBBMSNGDII2I2MT5IHK223AP2LD2RZLGMI2UFH7EVB6R3A, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:36.322690505 (320h11m34.874160966s) [noise=true , address=145.79.227.21:4474 , piece_id=PECKNOFRUZZIVY5XJ32WCP3CFBJ7LERMD6QFZZPLAXA3B47VNYBQ, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:36.322718137 (320h11m34.874188599s) [seg_pos={0 0}, upload=281977]: Segment reached optimal threshold of 13 pieces. Nov 22 16:15:36.323447516 (320h11m34.874917977s) [noise=true , address=145.79.227.11:4731 , piece_id=ASYJFIYGHG2VMQE5Y2W6J2MH4HE4TJX5HB36VDPNL6JI6LOBOWMA, seg_pos={0 0}, upload=281977]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9): protocol: rpc: dial tcp 145.79.227.11:4731: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 16:15:36.323459900 (320h11m34.874930371s) [noise=true , address=145.79.227.11:4731 , piece_id=ASYJFIYGHG2VMQE5Y2W6J2MH4HE4TJX5HB36VDPNL6JI6LOBOWMA, seg_pos={0 0}, upload=281977]: Long tail cancellation due to optimal threshold Nov 22 16:15:36.323615366 (320h11m34.875085817s) [noise=true , address=145.79.227.20:4437 , piece_id=6ION5L5UGDHAJQTYOAC34BTWDUJOJQXDF4XBRF7SQMRY57BYECOQ, seg_pos={0 0}, upload=281977]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1G3fe4SHMdNAjAmqLNyFb6qDKL94aAWGYE9qhczKmozKFhKKEV): protocol: write tcp 188.166.225.170:49846->145.79.227.20:4437: use of closed network connection; write tcp 188.166.225.170:49846->145.79.227.20:4437: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49846->145.79.227.20:4437: use of closed network connection Nov 22 16:15:36.323624343 (320h11m34.875094804s) [noise=true , address=145.79.227.20:4437 , piece_id=6ION5L5UGDHAJQTYOAC34BTWDUJOJQXDF4XBRF7SQMRY57BYECOQ, seg_pos={0 0}, upload=281977]: Long tail cancellation due to optimal threshold Nov 22 16:15:36.323867195 (320h11m34.875337677s) [noise=true , address=145.79.227.13:4802 , piece_id=VQ25RRLCCHN2BIEARHDMGO6JXOLM3RAAYSBGDJRZZAA7LU6XIORA, seg_pos={0 0}, upload=281977]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4): protocol: read tcp 188.166.225.170:59064->145.79.227.13:4802: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 16:15:36.323877184 (320h11m34.875347645s) [noise=true , address=145.79.227.13:4802 , piece_id=VQ25RRLCCHN2BIEARHDMGO6JXOLM3RAAYSBGDJRZZAA7LU6XIORA, seg_pos={0 0}, upload=281977]: Long tail cancellation due to optimal threshold Nov 22 16:15:36.323899958 (320h11m34.875370419s) [seg_pos={0 0}, upload=281977]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 22 16:15:36.324029074 (320h11m34.875499535s) [upload=281977]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 22 16:15:36.324036789 (320h11m34.87550726s) [upload=281977]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 22 16:15:36.620020293 (320h11m35.171490744s) [upload=281977]: Done uploading segments... Nov 22 16:15:36.620042956 (320h11m35.171513417s) [upload=281977]: Upload finished. err: <nil> Duration: 15.371338813s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 13 out of 60 is failed (threshold=6) [........XXX.XX.......X..XX......X.X...X.......X...........X.] duration: 3.481018123s trace: https://jaeger.dev.storj.io/trace/128554bcc962a445 FAIL (2025-11-22 16:34:49 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.481047811s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...............................XX...........................] duration: 14.240032479s trace: https://jaeger.dev.storj.io/trace/45889407e3b21f30 FAIL (2025-11-22 16:34:37 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ad Nov 22 16:15:10.157378989 (320h11m8.70884945s) [upload=281977]: Uploading segments... Nov 22 16:15:10.157518405 (320h11m8.708988866s) [upload=281977]: Got next segment. Inline: false Nov 22 16:15:10.157568901 (320h11m8.709039373s) [upload=281977]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 22 16:15:10.157576005 (320h11m8.709046466s) [upload=281977]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 22 16:15:10.422893065 (320h11m8.974363526s) [seg_pos={0 0}, upload=281977]: Begin upload segment... Nov 22 16:15:10.422954472 (320h11m8.974424933s) [seg_pos={0 0}, upload=281977]: Done begin upload segment. Nov 22 16:15:10.422989378 (320h11m8.974459839s) [noise=true , address=77.90.31.12:4589 , piece_id=DMJXC2AGYUS6IJDU57YMRVQ4XITS6C65KQPKPK77BAVNIWEDYW6A, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.423082535 (320h11m8.974553026s) [noise=true , address=145.79.227.11:4731 , piece_id=ASYJFIYGHG2VMQE5Y2W6J2MH4HE4TJX5HB36VDPNL6JI6LOBOWMA, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.423840959 (320h11m8.97531142s) [noise=true , address=45.91.116.4:4940 , piece_id=G7JVOOU24MGUUAJST54AWGAHJCQ3NEEDNYJCT7274VL3PMXXYEEA, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.424298210 (320h11m8.975768671s) [noise=true , address=145.79.227.28:4725 , piece_id=WAPK2NYLOGLXI4BOM6LA43UNUYKKVGCD6LA2KKPPIMXROR5AHCEA, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.424920144 (320h11m8.976390615s) [noise=true , address=145.79.227.14:4840 , piece_id=IFKFP5EBBMSNGDII2I2MT5IHK223AP2LD2RZLGMI2UFH7EVB6R3A, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.425382976 (320h11m8.976853437s) [noise=true , address=77.90.31.15:4699 , piece_id=6Q26DE27DV2TCYPSVMG64R36I4NJFZ6XMSX6ISYYJI6XD3E2TKQA, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.425904149 (320h11m8.97737461s) [noise=true , address=145.79.227.22:4509 , piece_id=XBHW2Q4ZMTXHM3DXV2L3H3UJRHW5NYMLO2Z5WMBPSR3QEKWM4YYA, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.426380896 (320h11m8.977851367s) [noise=true , address=145.79.227.13:4802 , piece_id=VQ25RRLCCHN2BIEARHDMGO6JXOLM3RAAYSBGDJRZZAA7LU6XIORA, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.427020925 (320h11m8.978491386s) [noise=true , address=145.79.227.21:4474 , piece_id=PECKNOFRUZZIVY5XJ32WCP3CFBJ7LERMD6QFZZPLAXA3B47VNYBQ, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.428223965 (320h11m8.979694426s) [noise=true , address=145.79.227.8:4331 , piece_id=IXCEZX2MCTDJJWMQY67WUQDLVKUBC6ILSZ66OTDRGZCF43DPPGSQ, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.430197163 (320h11m8.981667624s) [noise=true , address=77.90.31.12:4590 , piece_id=TUKND5MYHBQIH5FKBG6KIFDNAEUAGEX4SZ62D2BJFA77U2KU5UKQ, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.432639544 (320h11m8.984109995s) [noise=true , address=77.90.31.18:4807 , piece_id=QDVLUO3SEXUQJCCMJ4O5VNQK2XBTLXTYKKNZXDFTLXBFRUHMEKEQ, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.435162708 (320h11m8.986633169s) [noise=true , address=145.79.227.15:4874 , piece_id=FE5ZYSKFAKRZZ6RVLVKGZN46PBS5BZFZP5PKVYZNZFBXYGFIKFKA, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.437291813 (320h11m8.988762274s) [noise=true , address=145.79.227.6:4259 , piece_id=JFLTPOWA63S22QGQ6LN4WRFYVV2XQIDRLL7VGWFJRMCPXK5M6MXA, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.439344221 (320h11m8.990814682s) [noise=true , address=77.90.31.19:4841 , piece_id=VKYGMN3H4X2JJUFLNDGZZSCV5HCLQRLR53X5J36FGQ565MKWMNCA, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.440379501 (320h11m8.991849973s) [noise=true , address=145.79.227.20:4437 , piece_id=6ION5L5UGDHAJQTYOAC34BTWDUJOJQXDF4XBRF7SQMRY57BYECOQ, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:12.150193113 (320h11m10.701663574s) [upload=281977]: Next returned nil segment Nov 22 16:15:12.150240513 (320h11m10.701710974s) [upload=281977]: Waiting for error group managing segments... Nov 22 16:15:12.562473597 (320h11m11.113944048s) [noise=true , address=77.90.31.15:4699 , piece_id=6Q26DE27DV2TCYPSVMG64R36I4NJFZ6XMSX6ISYYJI6XD3E2TKQA, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:12.579741378 (320h11m11.131211839s) [noise=true , address=77.90.31.12:4590 , piece_id=TUKND5MYHBQIH5FKBG6KIFDNAEUAGEX4SZ62D2BJFA77U2KU5UKQ, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:12.586731719 (320h11m11.13820218s) [noise=true , address=45.91.116.4:4940 , piece_id=G7JVOOU24MGUUAJST54AWGAHJCQ3NEEDNYJCT7274VL3PMXXYEEA, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:12.598257075 (320h11m11.149727536s) [noise=true , address=77.90.31.12:4589 , piece_id=DMJXC2AGYUS6IJDU57YMRVQ4XITS6C65KQPKPK77BAVNIWEDYW6A, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:12.622577225 (320h11m11.174047686s) [noise=true , address=77.90.31.19:4841 , piece_id=VKYGMN3H4X2JJUFLNDGZZSCV5HCLQRLR53X5J36FGQ565MKWMNCA, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:12.754772838 (320h11m11.306243299s) [noise=true , address=77.90.31.18:4807 , piece_id=QDVLUO3SEXUQJCCMJ4O5VNQK2XBTLXTYKKNZXDFTLXBFRUHMEKEQ, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:19.382583530 (320h11m17.934053981s) [noise=true , address=145.79.227.15:4874 , piece_id=FE5ZYSKFAKRZZ6RVLVKGZN46PBS5BZFZP5PKVYZNZFBXYGFIKFKA, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:21.514760656 (320h11m20.066231117s) [noise=true , address=145.79.227.22:4509 , piece_id=XBHW2Q4ZMTXHM3DXV2L3H3UJRHW5NYMLO2Z5WMBPSR3QEKWM4YYA, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:23.636942539 (320h11m22.188413s) [noise=true , address=145.79.227.6:4259 , piece_id=JFLTPOWA63S22QGQ6LN4WRFYVV2XQIDRLL7VGWFJRMCPXK5M6MXA, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:23.872352010 (320h11m22.423822471s) [noise=true , address=145.79.227.28:4725 , piece_id=WAPK2NYLOGLXI4BOM6LA43UNUYKKVGCD6LA2KKPPIMXROR5AHCEA, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:25.966169160 (320h11m24.517639612s) [noise=true , address=145.79.227.8:4331 , piece_id=IXCEZX2MCTDJJWMQY67WUQDLVKUBC6ILSZ66OTDRGZCF43DPPGSQ, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:26.960865746 (320h11m25.512336207s) [noise=true , address=145.79.227.14:4840 , piece_id=IFKFP5EBBMSNGDII2I2MT5IHK223AP2LD2RZLGMI2UFH7EVB6R3A, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:36.322690505 (320h11m34.874160966s) [noise=true , address=145.79.227.21:4474 , piece_id=PECKNOFRUZZIVY5XJ32WCP3CFBJ7LERMD6QFZZPLAXA3B47VNYBQ, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:36.322718137 (320h11m34.874188599s) [seg_pos={0 0}, upload=281977]: Segment reached optimal threshold of 13 pieces. Nov 22 16:15:36.323447516 (320h11m34.874917977s) [noise=true , address=145.79.227.11:4731 , piece_id=ASYJFIYGHG2VMQE5Y2W6J2MH4HE4TJX5HB36VDPNL6JI6LOBOWMA, seg_pos={0 0}, upload=281977]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9): protocol: rpc: dial tcp 145.79.227.11:4731: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 16:15:36.323459900 (320h11m34.874930371s) [noise=true , address=145.79.227.11:4731 , piece_id=ASYJFIYGHG2VMQE5Y2W6J2MH4HE4TJX5HB36VDPNL6JI6LOBOWMA, seg_pos={0 0}, upload=281977]: Long tail cancellation due to optimal threshold Nov 22 16:15:36.323615366 (320h11m34.875085817s) [noise=true , address=145.79.227.20:4437 , piece_id=6ION5L5UGDHAJQTYOAC34BTWDUJOJQXDF4XBRF7SQMRY57BYECOQ, seg_pos={0 0}, upload=281977]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1G3fe4SHMdNAjAmqLNyFb6qDKL94aAWGYE9qhczKmozKFhKKEV): protocol: write tcp 188.166.225.170:49846->145.79.227.20:4437: use of closed network connection; write tcp 188.166.225.170:49846->145.79.227.20:4437: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49846->145.79.227.20:4437: use of closed network connection Nov 22 16:15:36.323624343 (320h11m34.875094804s) [noise=true , address=145.79.227.20:4437 , piece_id=6ION5L5UGDHAJQTYOAC34BTWDUJOJQXDF4XBRF7SQMRY57BYECOQ, seg_pos={0 0}, upload=281977]: Long tail cancellation due to optimal threshold Nov 22 16:15:36.323867195 (320h11m34.875337677s) [noise=true , address=145.79.227.13:4802 , piece_id=VQ25RRLCCHN2BIEARHDMGO6JXOLM3RAAYSBGDJRZZAA7LU6XIORA, seg_pos={0 0}, upload=281977]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4): protocol: read tcp 188.166.225.170:59064->145.79.227.13:4802: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 16:15:36.323877184 (320h11m34.875347645s) [noise=true , address=145.79.227.13:4802 , piece_id=VQ25RRLCCHN2BIEARHDMGO6JXOLM3RAAYSBGDJRZZAA7LU6XIORA, seg_pos={0 0}, upload=281977]: Long tail cancellation due to optimal threshold Nov 22 16:15:36.323899958 (320h11m34.875370419s) [seg_pos={0 0}, upload=281977]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 22 16:15:36.324029074 (320h11m34.875499535s) [upload=281977]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 22 16:15:36.324036789 (320h11m34.87550726s) [upload=281977]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 22 16:15:36.620020293 (320h11m35.171490744s) [upload=281977]: Done uploading segments... Nov 22 16:15:36.620042956 (320h11m35.171513417s) [upload=281977]: Upload finished. err: <nil> Duration: 14.240050525s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 13 out of 60 is failed (threshold=6) [.........XXX.XX.......X..XX......X.X...X.......X...........X] duration: 5.001360361s trace: https://jaeger.dev.storj.io/trace/128554bcc962a445 FAIL (2025-11-22 16:34:23 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001370921s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [................................XX..........................] duration: 13.75171953s trace: https://jaeger.dev.storj.io/trace/45889407e3b21f30 FAIL (2025-11-22 16:34:18 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ad Nov 22 16:15:10.157378989 (320h11m8.70884945s) [upload=281977]: Uploading segments... Nov 22 16:15:10.157518405 (320h11m8.708988866s) [upload=281977]: Got next segment. Inline: false Nov 22 16:15:10.157568901 (320h11m8.709039373s) [upload=281977]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 22 16:15:10.157576005 (320h11m8.709046466s) [upload=281977]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 22 16:15:10.422893065 (320h11m8.974363526s) [seg_pos={0 0}, upload=281977]: Begin upload segment... Nov 22 16:15:10.422954472 (320h11m8.974424933s) [seg_pos={0 0}, upload=281977]: Done begin upload segment. Nov 22 16:15:10.422989378 (320h11m8.974459839s) [noise=true , address=77.90.31.12:4589 , piece_id=DMJXC2AGYUS6IJDU57YMRVQ4XITS6C65KQPKPK77BAVNIWEDYW6A, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.423082535 (320h11m8.974553026s) [noise=true , address=145.79.227.11:4731 , piece_id=ASYJFIYGHG2VMQE5Y2W6J2MH4HE4TJX5HB36VDPNL6JI6LOBOWMA, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.423840959 (320h11m8.97531142s) [noise=true , address=45.91.116.4:4940 , piece_id=G7JVOOU24MGUUAJST54AWGAHJCQ3NEEDNYJCT7274VL3PMXXYEEA, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.424298210 (320h11m8.975768671s) [noise=true , address=145.79.227.28:4725 , piece_id=WAPK2NYLOGLXI4BOM6LA43UNUYKKVGCD6LA2KKPPIMXROR5AHCEA, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.424920144 (320h11m8.976390615s) [noise=true , address=145.79.227.14:4840 , piece_id=IFKFP5EBBMSNGDII2I2MT5IHK223AP2LD2RZLGMI2UFH7EVB6R3A, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.425382976 (320h11m8.976853437s) [noise=true , address=77.90.31.15:4699 , piece_id=6Q26DE27DV2TCYPSVMG64R36I4NJFZ6XMSX6ISYYJI6XD3E2TKQA, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.425904149 (320h11m8.97737461s) [noise=true , address=145.79.227.22:4509 , piece_id=XBHW2Q4ZMTXHM3DXV2L3H3UJRHW5NYMLO2Z5WMBPSR3QEKWM4YYA, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.426380896 (320h11m8.977851367s) [noise=true , address=145.79.227.13:4802 , piece_id=VQ25RRLCCHN2BIEARHDMGO6JXOLM3RAAYSBGDJRZZAA7LU6XIORA, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.427020925 (320h11m8.978491386s) [noise=true , address=145.79.227.21:4474 , piece_id=PECKNOFRUZZIVY5XJ32WCP3CFBJ7LERMD6QFZZPLAXA3B47VNYBQ, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.428223965 (320h11m8.979694426s) [noise=true , address=145.79.227.8:4331 , piece_id=IXCEZX2MCTDJJWMQY67WUQDLVKUBC6ILSZ66OTDRGZCF43DPPGSQ, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.430197163 (320h11m8.981667624s) [noise=true , address=77.90.31.12:4590 , piece_id=TUKND5MYHBQIH5FKBG6KIFDNAEUAGEX4SZ62D2BJFA77U2KU5UKQ, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.432639544 (320h11m8.984109995s) [noise=true , address=77.90.31.18:4807 , piece_id=QDVLUO3SEXUQJCCMJ4O5VNQK2XBTLXTYKKNZXDFTLXBFRUHMEKEQ, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.435162708 (320h11m8.986633169s) [noise=true , address=145.79.227.15:4874 , piece_id=FE5ZYSKFAKRZZ6RVLVKGZN46PBS5BZFZP5PKVYZNZFBXYGFIKFKA, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.437291813 (320h11m8.988762274s) [noise=true , address=145.79.227.6:4259 , piece_id=JFLTPOWA63S22QGQ6LN4WRFYVV2XQIDRLL7VGWFJRMCPXK5M6MXA, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.439344221 (320h11m8.990814682s) [noise=true , address=77.90.31.19:4841 , piece_id=VKYGMN3H4X2JJUFLNDGZZSCV5HCLQRLR53X5J36FGQ565MKWMNCA, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.440379501 (320h11m8.991849973s) [noise=true , address=145.79.227.20:4437 , piece_id=6ION5L5UGDHAJQTYOAC34BTWDUJOJQXDF4XBRF7SQMRY57BYECOQ, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:12.150193113 (320h11m10.701663574s) [upload=281977]: Next returned nil segment Nov 22 16:15:12.150240513 (320h11m10.701710974s) [upload=281977]: Waiting for error group managing segments... Nov 22 16:15:12.562473597 (320h11m11.113944048s) [noise=true , address=77.90.31.15:4699 , piece_id=6Q26DE27DV2TCYPSVMG64R36I4NJFZ6XMSX6ISYYJI6XD3E2TKQA, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:12.579741378 (320h11m11.131211839s) [noise=true , address=77.90.31.12:4590 , piece_id=TUKND5MYHBQIH5FKBG6KIFDNAEUAGEX4SZ62D2BJFA77U2KU5UKQ, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:12.586731719 (320h11m11.13820218s) [noise=true , address=45.91.116.4:4940 , piece_id=G7JVOOU24MGUUAJST54AWGAHJCQ3NEEDNYJCT7274VL3PMXXYEEA, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:12.598257075 (320h11m11.149727536s) [noise=true , address=77.90.31.12:4589 , piece_id=DMJXC2AGYUS6IJDU57YMRVQ4XITS6C65KQPKPK77BAVNIWEDYW6A, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:12.622577225 (320h11m11.174047686s) [noise=true , address=77.90.31.19:4841 , piece_id=VKYGMN3H4X2JJUFLNDGZZSCV5HCLQRLR53X5J36FGQ565MKWMNCA, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:12.754772838 (320h11m11.306243299s) [noise=true , address=77.90.31.18:4807 , piece_id=QDVLUO3SEXUQJCCMJ4O5VNQK2XBTLXTYKKNZXDFTLXBFRUHMEKEQ, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:19.382583530 (320h11m17.934053981s) [noise=true , address=145.79.227.15:4874 , piece_id=FE5ZYSKFAKRZZ6RVLVKGZN46PBS5BZFZP5PKVYZNZFBXYGFIKFKA, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:21.514760656 (320h11m20.066231117s) [noise=true , address=145.79.227.22:4509 , piece_id=XBHW2Q4ZMTXHM3DXV2L3H3UJRHW5NYMLO2Z5WMBPSR3QEKWM4YYA, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:23.636942539 (320h11m22.188413s) [noise=true , address=145.79.227.6:4259 , piece_id=JFLTPOWA63S22QGQ6LN4WRFYVV2XQIDRLL7VGWFJRMCPXK5M6MXA, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:23.872352010 (320h11m22.423822471s) [noise=true , address=145.79.227.28:4725 , piece_id=WAPK2NYLOGLXI4BOM6LA43UNUYKKVGCD6LA2KKPPIMXROR5AHCEA, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:25.966169160 (320h11m24.517639612s) [noise=true , address=145.79.227.8:4331 , piece_id=IXCEZX2MCTDJJWMQY67WUQDLVKUBC6ILSZ66OTDRGZCF43DPPGSQ, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:26.960865746 (320h11m25.512336207s) [noise=true , address=145.79.227.14:4840 , piece_id=IFKFP5EBBMSNGDII2I2MT5IHK223AP2LD2RZLGMI2UFH7EVB6R3A, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:36.322690505 (320h11m34.874160966s) [noise=true , address=145.79.227.21:4474 , piece_id=PECKNOFRUZZIVY5XJ32WCP3CFBJ7LERMD6QFZZPLAXA3B47VNYBQ, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:36.322718137 (320h11m34.874188599s) [seg_pos={0 0}, upload=281977]: Segment reached optimal threshold of 13 pieces. Nov 22 16:15:36.323447516 (320h11m34.874917977s) [noise=true , address=145.79.227.11:4731 , piece_id=ASYJFIYGHG2VMQE5Y2W6J2MH4HE4TJX5HB36VDPNL6JI6LOBOWMA, seg_pos={0 0}, upload=281977]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9): protocol: rpc: dial tcp 145.79.227.11:4731: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 16:15:36.323459900 (320h11m34.874930371s) [noise=true , address=145.79.227.11:4731 , piece_id=ASYJFIYGHG2VMQE5Y2W6J2MH4HE4TJX5HB36VDPNL6JI6LOBOWMA, seg_pos={0 0}, upload=281977]: Long tail cancellation due to optimal threshold Nov 22 16:15:36.323615366 (320h11m34.875085817s) [noise=true , address=145.79.227.20:4437 , piece_id=6ION5L5UGDHAJQTYOAC34BTWDUJOJQXDF4XBRF7SQMRY57BYECOQ, seg_pos={0 0}, upload=281977]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1G3fe4SHMdNAjAmqLNyFb6qDKL94aAWGYE9qhczKmozKFhKKEV): protocol: write tcp 188.166.225.170:49846->145.79.227.20:4437: use of closed network connection; write tcp 188.166.225.170:49846->145.79.227.20:4437: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49846->145.79.227.20:4437: use of closed network connection Nov 22 16:15:36.323624343 (320h11m34.875094804s) [noise=true , address=145.79.227.20:4437 , piece_id=6ION5L5UGDHAJQTYOAC34BTWDUJOJQXDF4XBRF7SQMRY57BYECOQ, seg_pos={0 0}, upload=281977]: Long tail cancellation due to optimal threshold Nov 22 16:15:36.323867195 (320h11m34.875337677s) [noise=true , address=145.79.227.13:4802 , piece_id=VQ25RRLCCHN2BIEARHDMGO6JXOLM3RAAYSBGDJRZZAA7LU6XIORA, seg_pos={0 0}, upload=281977]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4): protocol: read tcp 188.166.225.170:59064->145.79.227.13:4802: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 16:15:36.323877184 (320h11m34.875347645s) [noise=true , address=145.79.227.13:4802 , piece_id=VQ25RRLCCHN2BIEARHDMGO6JXOLM3RAAYSBGDJRZZAA7LU6XIORA, seg_pos={0 0}, upload=281977]: Long tail cancellation due to optimal threshold Nov 22 16:15:36.323899958 (320h11m34.875370419s) [seg_pos={0 0}, upload=281977]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 22 16:15:36.324029074 (320h11m34.875499535s) [upload=281977]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 22 16:15:36.324036789 (320h11m34.87550726s) [upload=281977]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 22 16:15:36.620020293 (320h11m35.171490744s) [upload=281977]: Done uploading segments... Nov 22 16:15:36.620042956 (320h11m35.171513417s) [upload=281977]: Upload finished. err: <nil> Duration: 13.751752501s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................X..................................] duration: 6.833878805s trace: https://jaeger.dev.storj.io/trace/13bcd3d57614c8f8 FAIL (2025-11-22 16:34:14 UTC) orbiter warning: context deadline exceeded Duration: 6.833906007s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................X.................................] duration: 12.982163273s trace: https://jaeger.dev.storj.io/trace/13bcd3d57614c8f8 FAIL (2025-11-22 16:33:56 UTC) orbiter warning: context deadline exceeded Duration: 12.982193972s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................X................................] duration: 7.670872189s trace: https://jaeger.dev.storj.io/trace/13bcd3d57614c8f8 FAIL (2025-11-22 16:33:28 UTC) orbiter warning: context deadline exceeded Duration: 7.67090425s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 12 out of 60 is failed (threshold=6) [..........XXX.XX.......X..XX......X.X...X.......X...........] duration: 3.55386563s trace: https://jaeger.dev.storj.io/trace/7237d130fff73497 FAIL (2025-11-22 16:33:26 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.553892922s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.................................XX.........................] duration: 15.34960141s trace: https://jaeger.dev.storj.io/trace/45889407e3b21f30 FAIL (2025-11-22 16:33:21 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ad Nov 22 16:15:10.157378989 (320h11m8.70884945s) [upload=281977]: Uploading segments... Nov 22 16:15:10.157518405 (320h11m8.708988866s) [upload=281977]: Got next segment. Inline: false Nov 22 16:15:10.157568901 (320h11m8.709039373s) [upload=281977]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 22 16:15:10.157576005 (320h11m8.709046466s) [upload=281977]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 22 16:15:10.422893065 (320h11m8.974363526s) [seg_pos={0 0}, upload=281977]: Begin upload segment... Nov 22 16:15:10.422954472 (320h11m8.974424933s) [seg_pos={0 0}, upload=281977]: Done begin upload segment. Nov 22 16:15:10.422989378 (320h11m8.974459839s) [noise=true , address=77.90.31.12:4589 , piece_id=DMJXC2AGYUS6IJDU57YMRVQ4XITS6C65KQPKPK77BAVNIWEDYW6A, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.423082535 (320h11m8.974553026s) [noise=true , address=145.79.227.11:4731 , piece_id=ASYJFIYGHG2VMQE5Y2W6J2MH4HE4TJX5HB36VDPNL6JI6LOBOWMA, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.423840959 (320h11m8.97531142s) [noise=true , address=45.91.116.4:4940 , piece_id=G7JVOOU24MGUUAJST54AWGAHJCQ3NEEDNYJCT7274VL3PMXXYEEA, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.424298210 (320h11m8.975768671s) [noise=true , address=145.79.227.28:4725 , piece_id=WAPK2NYLOGLXI4BOM6LA43UNUYKKVGCD6LA2KKPPIMXROR5AHCEA, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.424920144 (320h11m8.976390615s) [noise=true , address=145.79.227.14:4840 , piece_id=IFKFP5EBBMSNGDII2I2MT5IHK223AP2LD2RZLGMI2UFH7EVB6R3A, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.425382976 (320h11m8.976853437s) [noise=true , address=77.90.31.15:4699 , piece_id=6Q26DE27DV2TCYPSVMG64R36I4NJFZ6XMSX6ISYYJI6XD3E2TKQA, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.425904149 (320h11m8.97737461s) [noise=true , address=145.79.227.22:4509 , piece_id=XBHW2Q4ZMTXHM3DXV2L3H3UJRHW5NYMLO2Z5WMBPSR3QEKWM4YYA, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.426380896 (320h11m8.977851367s) [noise=true , address=145.79.227.13:4802 , piece_id=VQ25RRLCCHN2BIEARHDMGO6JXOLM3RAAYSBGDJRZZAA7LU6XIORA, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.427020925 (320h11m8.978491386s) [noise=true , address=145.79.227.21:4474 , piece_id=PECKNOFRUZZIVY5XJ32WCP3CFBJ7LERMD6QFZZPLAXA3B47VNYBQ, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.428223965 (320h11m8.979694426s) [noise=true , address=145.79.227.8:4331 , piece_id=IXCEZX2MCTDJJWMQY67WUQDLVKUBC6ILSZ66OTDRGZCF43DPPGSQ, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.430197163 (320h11m8.981667624s) [noise=true , address=77.90.31.12:4590 , piece_id=TUKND5MYHBQIH5FKBG6KIFDNAEUAGEX4SZ62D2BJFA77U2KU5UKQ, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.432639544 (320h11m8.984109995s) [noise=true , address=77.90.31.18:4807 , piece_id=QDVLUO3SEXUQJCCMJ4O5VNQK2XBTLXTYKKNZXDFTLXBFRUHMEKEQ, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.435162708 (320h11m8.986633169s) [noise=true , address=145.79.227.15:4874 , piece_id=FE5ZYSKFAKRZZ6RVLVKGZN46PBS5BZFZP5PKVYZNZFBXYGFIKFKA, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.437291813 (320h11m8.988762274s) [noise=true , address=145.79.227.6:4259 , piece_id=JFLTPOWA63S22QGQ6LN4WRFYVV2XQIDRLL7VGWFJRMCPXK5M6MXA, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.439344221 (320h11m8.990814682s) [noise=true , address=77.90.31.19:4841 , piece_id=VKYGMN3H4X2JJUFLNDGZZSCV5HCLQRLR53X5J36FGQ565MKWMNCA, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.440379501 (320h11m8.991849973s) [noise=true , address=145.79.227.20:4437 , piece_id=6ION5L5UGDHAJQTYOAC34BTWDUJOJQXDF4XBRF7SQMRY57BYECOQ, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:12.150193113 (320h11m10.701663574s) [upload=281977]: Next returned nil segment Nov 22 16:15:12.150240513 (320h11m10.701710974s) [upload=281977]: Waiting for error group managing segments... Nov 22 16:15:12.562473597 (320h11m11.113944048s) [noise=true , address=77.90.31.15:4699 , piece_id=6Q26DE27DV2TCYPSVMG64R36I4NJFZ6XMSX6ISYYJI6XD3E2TKQA, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:12.579741378 (320h11m11.131211839s) [noise=true , address=77.90.31.12:4590 , piece_id=TUKND5MYHBQIH5FKBG6KIFDNAEUAGEX4SZ62D2BJFA77U2KU5UKQ, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:12.586731719 (320h11m11.13820218s) [noise=true , address=45.91.116.4:4940 , piece_id=G7JVOOU24MGUUAJST54AWGAHJCQ3NEEDNYJCT7274VL3PMXXYEEA, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:12.598257075 (320h11m11.149727536s) [noise=true , address=77.90.31.12:4589 , piece_id=DMJXC2AGYUS6IJDU57YMRVQ4XITS6C65KQPKPK77BAVNIWEDYW6A, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:12.622577225 (320h11m11.174047686s) [noise=true , address=77.90.31.19:4841 , piece_id=VKYGMN3H4X2JJUFLNDGZZSCV5HCLQRLR53X5J36FGQ565MKWMNCA, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:12.754772838 (320h11m11.306243299s) [noise=true , address=77.90.31.18:4807 , piece_id=QDVLUO3SEXUQJCCMJ4O5VNQK2XBTLXTYKKNZXDFTLXBFRUHMEKEQ, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:19.382583530 (320h11m17.934053981s) [noise=true , address=145.79.227.15:4874 , piece_id=FE5ZYSKFAKRZZ6RVLVKGZN46PBS5BZFZP5PKVYZNZFBXYGFIKFKA, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:21.514760656 (320h11m20.066231117s) [noise=true , address=145.79.227.22:4509 , piece_id=XBHW2Q4ZMTXHM3DXV2L3H3UJRHW5NYMLO2Z5WMBPSR3QEKWM4YYA, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:23.636942539 (320h11m22.188413s) [noise=true , address=145.79.227.6:4259 , piece_id=JFLTPOWA63S22QGQ6LN4WRFYVV2XQIDRLL7VGWFJRMCPXK5M6MXA, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:23.872352010 (320h11m22.423822471s) [noise=true , address=145.79.227.28:4725 , piece_id=WAPK2NYLOGLXI4BOM6LA43UNUYKKVGCD6LA2KKPPIMXROR5AHCEA, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:25.966169160 (320h11m24.517639612s) [noise=true , address=145.79.227.8:4331 , piece_id=IXCEZX2MCTDJJWMQY67WUQDLVKUBC6ILSZ66OTDRGZCF43DPPGSQ, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:26.960865746 (320h11m25.512336207s) [noise=true , address=145.79.227.14:4840 , piece_id=IFKFP5EBBMSNGDII2I2MT5IHK223AP2LD2RZLGMI2UFH7EVB6R3A, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:36.322690505 (320h11m34.874160966s) [noise=true , address=145.79.227.21:4474 , piece_id=PECKNOFRUZZIVY5XJ32WCP3CFBJ7LERMD6QFZZPLAXA3B47VNYBQ, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:36.322718137 (320h11m34.874188599s) [seg_pos={0 0}, upload=281977]: Segment reached optimal threshold of 13 pieces. Nov 22 16:15:36.323447516 (320h11m34.874917977s) [noise=true , address=145.79.227.11:4731 , piece_id=ASYJFIYGHG2VMQE5Y2W6J2MH4HE4TJX5HB36VDPNL6JI6LOBOWMA, seg_pos={0 0}, upload=281977]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9): protocol: rpc: dial tcp 145.79.227.11:4731: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 16:15:36.323459900 (320h11m34.874930371s) [noise=true , address=145.79.227.11:4731 , piece_id=ASYJFIYGHG2VMQE5Y2W6J2MH4HE4TJX5HB36VDPNL6JI6LOBOWMA, seg_pos={0 0}, upload=281977]: Long tail cancellation due to optimal threshold Nov 22 16:15:36.323615366 (320h11m34.875085817s) [noise=true , address=145.79.227.20:4437 , piece_id=6ION5L5UGDHAJQTYOAC34BTWDUJOJQXDF4XBRF7SQMRY57BYECOQ, seg_pos={0 0}, upload=281977]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1G3fe4SHMdNAjAmqLNyFb6qDKL94aAWGYE9qhczKmozKFhKKEV): protocol: write tcp 188.166.225.170:49846->145.79.227.20:4437: use of closed network connection; write tcp 188.166.225.170:49846->145.79.227.20:4437: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49846->145.79.227.20:4437: use of closed network connection Nov 22 16:15:36.323624343 (320h11m34.875094804s) [noise=true , address=145.79.227.20:4437 , piece_id=6ION5L5UGDHAJQTYOAC34BTWDUJOJQXDF4XBRF7SQMRY57BYECOQ, seg_pos={0 0}, upload=281977]: Long tail cancellation due to optimal threshold Nov 22 16:15:36.323867195 (320h11m34.875337677s) [noise=true , address=145.79.227.13:4802 , piece_id=VQ25RRLCCHN2BIEARHDMGO6JXOLM3RAAYSBGDJRZZAA7LU6XIORA, seg_pos={0 0}, upload=281977]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4): protocol: read tcp 188.166.225.170:59064->145.79.227.13:4802: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 16:15:36.323877184 (320h11m34.875347645s) [noise=true , address=145.79.227.13:4802 , piece_id=VQ25RRLCCHN2BIEARHDMGO6JXOLM3RAAYSBGDJRZZAA7LU6XIORA, seg_pos={0 0}, upload=281977]: Long tail cancellation due to optimal threshold Nov 22 16:15:36.323899958 (320h11m34.875370419s) [seg_pos={0 0}, upload=281977]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 22 16:15:36.324029074 (320h11m34.875499535s) [upload=281977]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 22 16:15:36.324036789 (320h11m34.87550726s) [upload=281977]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 22 16:15:36.620020293 (320h11m35.171490744s) [upload=281977]: Done uploading segments... Nov 22 16:15:36.620042956 (320h11m35.171513417s) [upload=281977]: Upload finished. err: <nil> Duration: 15.349632268s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 12 out of 60 is failed (threshold=6) [...........XXX.XX.......X..XX......X.X...X.......X..........] duration: 3.484904576s trace: https://jaeger.dev.storj.io/trace/7237d130fff73497 FAIL (2025-11-22 16:33:05 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.48493835s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................X...............................] duration: 6.698187777s trace: https://jaeger.dev.storj.io/trace/13bcd3d57614c8f8 FAIL (2025-11-22 16:32:44 UTC) orbiter warning: context deadline exceeded Duration: 6.698214298s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 12 out of 60 is failed (threshold=6) [............XXX.XX.......X..XX......X.X...X.......X.........] duration: 3.632509061s trace: https://jaeger.dev.storj.io/trace/7237d130fff73497 FAIL (2025-11-22 16:32:44 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.632536633s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..................................XX........................] duration: 15.087324448s trace: https://jaeger.dev.storj.io/trace/45889407e3b21f30 FAIL (2025-11-22 16:32:26 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ad Nov 22 16:15:10.157378989 (320h11m8.70884945s) [upload=281977]: Uploading segments... Nov 22 16:15:10.157518405 (320h11m8.708988866s) [upload=281977]: Got next segment. Inline: false Nov 22 16:15:10.157568901 (320h11m8.709039373s) [upload=281977]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 22 16:15:10.157576005 (320h11m8.709046466s) [upload=281977]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 22 16:15:10.422893065 (320h11m8.974363526s) [seg_pos={0 0}, upload=281977]: Begin upload segment... Nov 22 16:15:10.422954472 (320h11m8.974424933s) [seg_pos={0 0}, upload=281977]: Done begin upload segment. Nov 22 16:15:10.422989378 (320h11m8.974459839s) [noise=true , address=77.90.31.12:4589 , piece_id=DMJXC2AGYUS6IJDU57YMRVQ4XITS6C65KQPKPK77BAVNIWEDYW6A, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.423082535 (320h11m8.974553026s) [noise=true , address=145.79.227.11:4731 , piece_id=ASYJFIYGHG2VMQE5Y2W6J2MH4HE4TJX5HB36VDPNL6JI6LOBOWMA, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.423840959 (320h11m8.97531142s) [noise=true , address=45.91.116.4:4940 , piece_id=G7JVOOU24MGUUAJST54AWGAHJCQ3NEEDNYJCT7274VL3PMXXYEEA, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.424298210 (320h11m8.975768671s) [noise=true , address=145.79.227.28:4725 , piece_id=WAPK2NYLOGLXI4BOM6LA43UNUYKKVGCD6LA2KKPPIMXROR5AHCEA, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.424920144 (320h11m8.976390615s) [noise=true , address=145.79.227.14:4840 , piece_id=IFKFP5EBBMSNGDII2I2MT5IHK223AP2LD2RZLGMI2UFH7EVB6R3A, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.425382976 (320h11m8.976853437s) [noise=true , address=77.90.31.15:4699 , piece_id=6Q26DE27DV2TCYPSVMG64R36I4NJFZ6XMSX6ISYYJI6XD3E2TKQA, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.425904149 (320h11m8.97737461s) [noise=true , address=145.79.227.22:4509 , piece_id=XBHW2Q4ZMTXHM3DXV2L3H3UJRHW5NYMLO2Z5WMBPSR3QEKWM4YYA, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.426380896 (320h11m8.977851367s) [noise=true , address=145.79.227.13:4802 , piece_id=VQ25RRLCCHN2BIEARHDMGO6JXOLM3RAAYSBGDJRZZAA7LU6XIORA, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.427020925 (320h11m8.978491386s) [noise=true , address=145.79.227.21:4474 , piece_id=PECKNOFRUZZIVY5XJ32WCP3CFBJ7LERMD6QFZZPLAXA3B47VNYBQ, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.428223965 (320h11m8.979694426s) [noise=true , address=145.79.227.8:4331 , piece_id=IXCEZX2MCTDJJWMQY67WUQDLVKUBC6ILSZ66OTDRGZCF43DPPGSQ, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.430197163 (320h11m8.981667624s) [noise=true , address=77.90.31.12:4590 , piece_id=TUKND5MYHBQIH5FKBG6KIFDNAEUAGEX4SZ62D2BJFA77U2KU5UKQ, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.432639544 (320h11m8.984109995s) [noise=true , address=77.90.31.18:4807 , piece_id=QDVLUO3SEXUQJCCMJ4O5VNQK2XBTLXTYKKNZXDFTLXBFRUHMEKEQ, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.435162708 (320h11m8.986633169s) [noise=true , address=145.79.227.15:4874 , piece_id=FE5ZYSKFAKRZZ6RVLVKGZN46PBS5BZFZP5PKVYZNZFBXYGFIKFKA, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.437291813 (320h11m8.988762274s) [noise=true , address=145.79.227.6:4259 , piece_id=JFLTPOWA63S22QGQ6LN4WRFYVV2XQIDRLL7VGWFJRMCPXK5M6MXA, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.439344221 (320h11m8.990814682s) [noise=true , address=77.90.31.19:4841 , piece_id=VKYGMN3H4X2JJUFLNDGZZSCV5HCLQRLR53X5J36FGQ565MKWMNCA, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.440379501 (320h11m8.991849973s) [noise=true , address=145.79.227.20:4437 , piece_id=6ION5L5UGDHAJQTYOAC34BTWDUJOJQXDF4XBRF7SQMRY57BYECOQ, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:12.150193113 (320h11m10.701663574s) [upload=281977]: Next returned nil segment Nov 22 16:15:12.150240513 (320h11m10.701710974s) [upload=281977]: Waiting for error group managing segments... Nov 22 16:15:12.562473597 (320h11m11.113944048s) [noise=true , address=77.90.31.15:4699 , piece_id=6Q26DE27DV2TCYPSVMG64R36I4NJFZ6XMSX6ISYYJI6XD3E2TKQA, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:12.579741378 (320h11m11.131211839s) [noise=true , address=77.90.31.12:4590 , piece_id=TUKND5MYHBQIH5FKBG6KIFDNAEUAGEX4SZ62D2BJFA77U2KU5UKQ, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:12.586731719 (320h11m11.13820218s) [noise=true , address=45.91.116.4:4940 , piece_id=G7JVOOU24MGUUAJST54AWGAHJCQ3NEEDNYJCT7274VL3PMXXYEEA, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:12.598257075 (320h11m11.149727536s) [noise=true , address=77.90.31.12:4589 , piece_id=DMJXC2AGYUS6IJDU57YMRVQ4XITS6C65KQPKPK77BAVNIWEDYW6A, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:12.622577225 (320h11m11.174047686s) [noise=true , address=77.90.31.19:4841 , piece_id=VKYGMN3H4X2JJUFLNDGZZSCV5HCLQRLR53X5J36FGQ565MKWMNCA, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:12.754772838 (320h11m11.306243299s) [noise=true , address=77.90.31.18:4807 , piece_id=QDVLUO3SEXUQJCCMJ4O5VNQK2XBTLXTYKKNZXDFTLXBFRUHMEKEQ, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:19.382583530 (320h11m17.934053981s) [noise=true , address=145.79.227.15:4874 , piece_id=FE5ZYSKFAKRZZ6RVLVKGZN46PBS5BZFZP5PKVYZNZFBXYGFIKFKA, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:21.514760656 (320h11m20.066231117s) [noise=true , address=145.79.227.22:4509 , piece_id=XBHW2Q4ZMTXHM3DXV2L3H3UJRHW5NYMLO2Z5WMBPSR3QEKWM4YYA, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:23.636942539 (320h11m22.188413s) [noise=true , address=145.79.227.6:4259 , piece_id=JFLTPOWA63S22QGQ6LN4WRFYVV2XQIDRLL7VGWFJRMCPXK5M6MXA, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:23.872352010 (320h11m22.423822471s) [noise=true , address=145.79.227.28:4725 , piece_id=WAPK2NYLOGLXI4BOM6LA43UNUYKKVGCD6LA2KKPPIMXROR5AHCEA, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:25.966169160 (320h11m24.517639612s) [noise=true , address=145.79.227.8:4331 , piece_id=IXCEZX2MCTDJJWMQY67WUQDLVKUBC6ILSZ66OTDRGZCF43DPPGSQ, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:26.960865746 (320h11m25.512336207s) [noise=true , address=145.79.227.14:4840 , piece_id=IFKFP5EBBMSNGDII2I2MT5IHK223AP2LD2RZLGMI2UFH7EVB6R3A, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:36.322690505 (320h11m34.874160966s) [noise=true , address=145.79.227.21:4474 , piece_id=PECKNOFRUZZIVY5XJ32WCP3CFBJ7LERMD6QFZZPLAXA3B47VNYBQ, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:36.322718137 (320h11m34.874188599s) [seg_pos={0 0}, upload=281977]: Segment reached optimal threshold of 13 pieces. Nov 22 16:15:36.323447516 (320h11m34.874917977s) [noise=true , address=145.79.227.11:4731 , piece_id=ASYJFIYGHG2VMQE5Y2W6J2MH4HE4TJX5HB36VDPNL6JI6LOBOWMA, seg_pos={0 0}, upload=281977]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9): protocol: rpc: dial tcp 145.79.227.11:4731: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 16:15:36.323459900 (320h11m34.874930371s) [noise=true , address=145.79.227.11:4731 , piece_id=ASYJFIYGHG2VMQE5Y2W6J2MH4HE4TJX5HB36VDPNL6JI6LOBOWMA, seg_pos={0 0}, upload=281977]: Long tail cancellation due to optimal threshold Nov 22 16:15:36.323615366 (320h11m34.875085817s) [noise=true , address=145.79.227.20:4437 , piece_id=6ION5L5UGDHAJQTYOAC34BTWDUJOJQXDF4XBRF7SQMRY57BYECOQ, seg_pos={0 0}, upload=281977]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1G3fe4SHMdNAjAmqLNyFb6qDKL94aAWGYE9qhczKmozKFhKKEV): protocol: write tcp 188.166.225.170:49846->145.79.227.20:4437: use of closed network connection; write tcp 188.166.225.170:49846->145.79.227.20:4437: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49846->145.79.227.20:4437: use of closed network connection Nov 22 16:15:36.323624343 (320h11m34.875094804s) [noise=true , address=145.79.227.20:4437 , piece_id=6ION5L5UGDHAJQTYOAC34BTWDUJOJQXDF4XBRF7SQMRY57BYECOQ, seg_pos={0 0}, upload=281977]: Long tail cancellation due to optimal threshold Nov 22 16:15:36.323867195 (320h11m34.875337677s) [noise=true , address=145.79.227.13:4802 , piece_id=VQ25RRLCCHN2BIEARHDMGO6JXOLM3RAAYSBGDJRZZAA7LU6XIORA, seg_pos={0 0}, upload=281977]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4): protocol: read tcp 188.166.225.170:59064->145.79.227.13:4802: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 16:15:36.323877184 (320h11m34.875347645s) [noise=true , address=145.79.227.13:4802 , piece_id=VQ25RRLCCHN2BIEARHDMGO6JXOLM3RAAYSBGDJRZZAA7LU6XIORA, seg_pos={0 0}, upload=281977]: Long tail cancellation due to optimal threshold Nov 22 16:15:36.323899958 (320h11m34.875370419s) [seg_pos={0 0}, upload=281977]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 22 16:15:36.324029074 (320h11m34.875499535s) [upload=281977]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 22 16:15:36.324036789 (320h11m34.87550726s) [upload=281977]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 22 16:15:36.620020293 (320h11m35.171490744s) [upload=281977]: Done uploading segments... Nov 22 16:15:36.620042956 (320h11m35.171513417s) [upload=281977]: Upload finished. err: <nil> Duration: 15.087350187s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 12 out of 60 is failed (threshold=6) [.............XXX.XX.......X..XX......X.X...X.......X........] duration: 3.5260271s trace: https://jaeger.dev.storj.io/trace/7237d130fff73497 FAIL (2025-11-22 16:32:24 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.526074471s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................X..............................] duration: 13.846655068s trace: https://jaeger.dev.storj.io/trace/13bcd3d57614c8f8 FAIL (2025-11-22 16:31:57 UTC) orbiter warning: context deadline exceeded Duration: 13.846676158s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 12 out of 60 is failed (threshold=6) [..............XXX.XX.......X..XX......X.X...X.......X.......] duration: 4.523476028s trace: https://jaeger.dev.storj.io/trace/7237d130fff73497 FAIL (2025-11-22 16:31:52 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.523507748s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...................................XX.......................] duration: 14.653175362s trace: https://jaeger.dev.storj.io/trace/45889407e3b21f30 FAIL (2025-11-22 16:31:35 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ad Nov 22 16:15:10.157378989 (320h11m8.70884945s) [upload=281977]: Uploading segments... Nov 22 16:15:10.157518405 (320h11m8.708988866s) [upload=281977]: Got next segment. Inline: false Nov 22 16:15:10.157568901 (320h11m8.709039373s) [upload=281977]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 22 16:15:10.157576005 (320h11m8.709046466s) [upload=281977]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 22 16:15:10.422893065 (320h11m8.974363526s) [seg_pos={0 0}, upload=281977]: Begin upload segment... Nov 22 16:15:10.422954472 (320h11m8.974424933s) [seg_pos={0 0}, upload=281977]: Done begin upload segment. Nov 22 16:15:10.422989378 (320h11m8.974459839s) [noise=true , address=77.90.31.12:4589 , piece_id=DMJXC2AGYUS6IJDU57YMRVQ4XITS6C65KQPKPK77BAVNIWEDYW6A, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.423082535 (320h11m8.974553026s) [noise=true , address=145.79.227.11:4731 , piece_id=ASYJFIYGHG2VMQE5Y2W6J2MH4HE4TJX5HB36VDPNL6JI6LOBOWMA, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.423840959 (320h11m8.97531142s) [noise=true , address=45.91.116.4:4940 , piece_id=G7JVOOU24MGUUAJST54AWGAHJCQ3NEEDNYJCT7274VL3PMXXYEEA, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.424298210 (320h11m8.975768671s) [noise=true , address=145.79.227.28:4725 , piece_id=WAPK2NYLOGLXI4BOM6LA43UNUYKKVGCD6LA2KKPPIMXROR5AHCEA, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.424920144 (320h11m8.976390615s) [noise=true , address=145.79.227.14:4840 , piece_id=IFKFP5EBBMSNGDII2I2MT5IHK223AP2LD2RZLGMI2UFH7EVB6R3A, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.425382976 (320h11m8.976853437s) [noise=true , address=77.90.31.15:4699 , piece_id=6Q26DE27DV2TCYPSVMG64R36I4NJFZ6XMSX6ISYYJI6XD3E2TKQA, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.425904149 (320h11m8.97737461s) [noise=true , address=145.79.227.22:4509 , piece_id=XBHW2Q4ZMTXHM3DXV2L3H3UJRHW5NYMLO2Z5WMBPSR3QEKWM4YYA, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.426380896 (320h11m8.977851367s) [noise=true , address=145.79.227.13:4802 , piece_id=VQ25RRLCCHN2BIEARHDMGO6JXOLM3RAAYSBGDJRZZAA7LU6XIORA, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.427020925 (320h11m8.978491386s) [noise=true , address=145.79.227.21:4474 , piece_id=PECKNOFRUZZIVY5XJ32WCP3CFBJ7LERMD6QFZZPLAXA3B47VNYBQ, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.428223965 (320h11m8.979694426s) [noise=true , address=145.79.227.8:4331 , piece_id=IXCEZX2MCTDJJWMQY67WUQDLVKUBC6ILSZ66OTDRGZCF43DPPGSQ, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.430197163 (320h11m8.981667624s) [noise=true , address=77.90.31.12:4590 , piece_id=TUKND5MYHBQIH5FKBG6KIFDNAEUAGEX4SZ62D2BJFA77U2KU5UKQ, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.432639544 (320h11m8.984109995s) [noise=true , address=77.90.31.18:4807 , piece_id=QDVLUO3SEXUQJCCMJ4O5VNQK2XBTLXTYKKNZXDFTLXBFRUHMEKEQ, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.435162708 (320h11m8.986633169s) [noise=true , address=145.79.227.15:4874 , piece_id=FE5ZYSKFAKRZZ6RVLVKGZN46PBS5BZFZP5PKVYZNZFBXYGFIKFKA, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.437291813 (320h11m8.988762274s) [noise=true , address=145.79.227.6:4259 , piece_id=JFLTPOWA63S22QGQ6LN4WRFYVV2XQIDRLL7VGWFJRMCPXK5M6MXA, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.439344221 (320h11m8.990814682s) [noise=true , address=77.90.31.19:4841 , piece_id=VKYGMN3H4X2JJUFLNDGZZSCV5HCLQRLR53X5J36FGQ565MKWMNCA, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.440379501 (320h11m8.991849973s) [noise=true , address=145.79.227.20:4437 , piece_id=6ION5L5UGDHAJQTYOAC34BTWDUJOJQXDF4XBRF7SQMRY57BYECOQ, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:12.150193113 (320h11m10.701663574s) [upload=281977]: Next returned nil segment Nov 22 16:15:12.150240513 (320h11m10.701710974s) [upload=281977]: Waiting for error group managing segments... Nov 22 16:15:12.562473597 (320h11m11.113944048s) [noise=true , address=77.90.31.15:4699 , piece_id=6Q26DE27DV2TCYPSVMG64R36I4NJFZ6XMSX6ISYYJI6XD3E2TKQA, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:12.579741378 (320h11m11.131211839s) [noise=true , address=77.90.31.12:4590 , piece_id=TUKND5MYHBQIH5FKBG6KIFDNAEUAGEX4SZ62D2BJFA77U2KU5UKQ, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:12.586731719 (320h11m11.13820218s) [noise=true , address=45.91.116.4:4940 , piece_id=G7JVOOU24MGUUAJST54AWGAHJCQ3NEEDNYJCT7274VL3PMXXYEEA, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:12.598257075 (320h11m11.149727536s) [noise=true , address=77.90.31.12:4589 , piece_id=DMJXC2AGYUS6IJDU57YMRVQ4XITS6C65KQPKPK77BAVNIWEDYW6A, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:12.622577225 (320h11m11.174047686s) [noise=true , address=77.90.31.19:4841 , piece_id=VKYGMN3H4X2JJUFLNDGZZSCV5HCLQRLR53X5J36FGQ565MKWMNCA, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:12.754772838 (320h11m11.306243299s) [noise=true , address=77.90.31.18:4807 , piece_id=QDVLUO3SEXUQJCCMJ4O5VNQK2XBTLXTYKKNZXDFTLXBFRUHMEKEQ, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:19.382583530 (320h11m17.934053981s) [noise=true , address=145.79.227.15:4874 , piece_id=FE5ZYSKFAKRZZ6RVLVKGZN46PBS5BZFZP5PKVYZNZFBXYGFIKFKA, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:21.514760656 (320h11m20.066231117s) [noise=true , address=145.79.227.22:4509 , piece_id=XBHW2Q4ZMTXHM3DXV2L3H3UJRHW5NYMLO2Z5WMBPSR3QEKWM4YYA, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:23.636942539 (320h11m22.188413s) [noise=true , address=145.79.227.6:4259 , piece_id=JFLTPOWA63S22QGQ6LN4WRFYVV2XQIDRLL7VGWFJRMCPXK5M6MXA, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:23.872352010 (320h11m22.423822471s) [noise=true , address=145.79.227.28:4725 , piece_id=WAPK2NYLOGLXI4BOM6LA43UNUYKKVGCD6LA2KKPPIMXROR5AHCEA, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:25.966169160 (320h11m24.517639612s) [noise=true , address=145.79.227.8:4331 , piece_id=IXCEZX2MCTDJJWMQY67WUQDLVKUBC6ILSZ66OTDRGZCF43DPPGSQ, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:26.960865746 (320h11m25.512336207s) [noise=true , address=145.79.227.14:4840 , piece_id=IFKFP5EBBMSNGDII2I2MT5IHK223AP2LD2RZLGMI2UFH7EVB6R3A, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:36.322690505 (320h11m34.874160966s) [noise=true , address=145.79.227.21:4474 , piece_id=PECKNOFRUZZIVY5XJ32WCP3CFBJ7LERMD6QFZZPLAXA3B47VNYBQ, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:36.322718137 (320h11m34.874188599s) [seg_pos={0 0}, upload=281977]: Segment reached optimal threshold of 13 pieces. Nov 22 16:15:36.323447516 (320h11m34.874917977s) [noise=true , address=145.79.227.11:4731 , piece_id=ASYJFIYGHG2VMQE5Y2W6J2MH4HE4TJX5HB36VDPNL6JI6LOBOWMA, seg_pos={0 0}, upload=281977]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9): protocol: rpc: dial tcp 145.79.227.11:4731: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 16:15:36.323459900 (320h11m34.874930371s) [noise=true , address=145.79.227.11:4731 , piece_id=ASYJFIYGHG2VMQE5Y2W6J2MH4HE4TJX5HB36VDPNL6JI6LOBOWMA, seg_pos={0 0}, upload=281977]: Long tail cancellation due to optimal threshold Nov 22 16:15:36.323615366 (320h11m34.875085817s) [noise=true , address=145.79.227.20:4437 , piece_id=6ION5L5UGDHAJQTYOAC34BTWDUJOJQXDF4XBRF7SQMRY57BYECOQ, seg_pos={0 0}, upload=281977]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1G3fe4SHMdNAjAmqLNyFb6qDKL94aAWGYE9qhczKmozKFhKKEV): protocol: write tcp 188.166.225.170:49846->145.79.227.20:4437: use of closed network connection; write tcp 188.166.225.170:49846->145.79.227.20:4437: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49846->145.79.227.20:4437: use of closed network connection Nov 22 16:15:36.323624343 (320h11m34.875094804s) [noise=true , address=145.79.227.20:4437 , piece_id=6ION5L5UGDHAJQTYOAC34BTWDUJOJQXDF4XBRF7SQMRY57BYECOQ, seg_pos={0 0}, upload=281977]: Long tail cancellation due to optimal threshold Nov 22 16:15:36.323867195 (320h11m34.875337677s) [noise=true , address=145.79.227.13:4802 , piece_id=VQ25RRLCCHN2BIEARHDMGO6JXOLM3RAAYSBGDJRZZAA7LU6XIORA, seg_pos={0 0}, upload=281977]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4): protocol: read tcp 188.166.225.170:59064->145.79.227.13:4802: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 16:15:36.323877184 (320h11m34.875347645s) [noise=true , address=145.79.227.13:4802 , piece_id=VQ25RRLCCHN2BIEARHDMGO6JXOLM3RAAYSBGDJRZZAA7LU6XIORA, seg_pos={0 0}, upload=281977]: Long tail cancellation due to optimal threshold Nov 22 16:15:36.323899958 (320h11m34.875370419s) [seg_pos={0 0}, upload=281977]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 22 16:15:36.324029074 (320h11m34.875499535s) [upload=281977]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 22 16:15:36.324036789 (320h11m34.87550726s) [upload=281977]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 22 16:15:36.620020293 (320h11m35.171490744s) [upload=281977]: Done uploading segments... Nov 22 16:15:36.620042956 (320h11m35.171513417s) [upload=281977]: Upload finished. err: <nil> Duration: 14.653216061s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 12 out of 60 is failed (threshold=6) [...............XXX.XX.......X..XX......X.X...X.......X......] duration: 3.628245213s trace: https://jaeger.dev.storj.io/trace/7237d130fff73497 FAIL (2025-11-22 16:31:04 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.628272935s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................X.............................] duration: 17.877489959s trace: https://jaeger.dev.storj.io/trace/13bcd3d57614c8f8 FAIL (2025-11-22 16:31:01 UTC) orbiter warning: context deadline exceeded Duration: 17.877651447s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [....................................XX......................] duration: 14.875771525s trace: https://jaeger.dev.storj.io/trace/45889407e3b21f30 FAIL (2025-11-22 16:31:00 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ad Nov 22 16:15:10.157378989 (320h11m8.70884945s) [upload=281977]: Uploading segments... Nov 22 16:15:10.157518405 (320h11m8.708988866s) [upload=281977]: Got next segment. Inline: false Nov 22 16:15:10.157568901 (320h11m8.709039373s) [upload=281977]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 22 16:15:10.157576005 (320h11m8.709046466s) [upload=281977]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 22 16:15:10.422893065 (320h11m8.974363526s) [seg_pos={0 0}, upload=281977]: Begin upload segment... Nov 22 16:15:10.422954472 (320h11m8.974424933s) [seg_pos={0 0}, upload=281977]: Done begin upload segment. Nov 22 16:15:10.422989378 (320h11m8.974459839s) [noise=true , address=77.90.31.12:4589 , piece_id=DMJXC2AGYUS6IJDU57YMRVQ4XITS6C65KQPKPK77BAVNIWEDYW6A, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.423082535 (320h11m8.974553026s) [noise=true , address=145.79.227.11:4731 , piece_id=ASYJFIYGHG2VMQE5Y2W6J2MH4HE4TJX5HB36VDPNL6JI6LOBOWMA, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.423840959 (320h11m8.97531142s) [noise=true , address=45.91.116.4:4940 , piece_id=G7JVOOU24MGUUAJST54AWGAHJCQ3NEEDNYJCT7274VL3PMXXYEEA, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.424298210 (320h11m8.975768671s) [noise=true , address=145.79.227.28:4725 , piece_id=WAPK2NYLOGLXI4BOM6LA43UNUYKKVGCD6LA2KKPPIMXROR5AHCEA, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.424920144 (320h11m8.976390615s) [noise=true , address=145.79.227.14:4840 , piece_id=IFKFP5EBBMSNGDII2I2MT5IHK223AP2LD2RZLGMI2UFH7EVB6R3A, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.425382976 (320h11m8.976853437s) [noise=true , address=77.90.31.15:4699 , piece_id=6Q26DE27DV2TCYPSVMG64R36I4NJFZ6XMSX6ISYYJI6XD3E2TKQA, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.425904149 (320h11m8.97737461s) [noise=true , address=145.79.227.22:4509 , piece_id=XBHW2Q4ZMTXHM3DXV2L3H3UJRHW5NYMLO2Z5WMBPSR3QEKWM4YYA, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.426380896 (320h11m8.977851367s) [noise=true , address=145.79.227.13:4802 , piece_id=VQ25RRLCCHN2BIEARHDMGO6JXOLM3RAAYSBGDJRZZAA7LU6XIORA, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.427020925 (320h11m8.978491386s) [noise=true , address=145.79.227.21:4474 , piece_id=PECKNOFRUZZIVY5XJ32WCP3CFBJ7LERMD6QFZZPLAXA3B47VNYBQ, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.428223965 (320h11m8.979694426s) [noise=true , address=145.79.227.8:4331 , piece_id=IXCEZX2MCTDJJWMQY67WUQDLVKUBC6ILSZ66OTDRGZCF43DPPGSQ, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.430197163 (320h11m8.981667624s) [noise=true , address=77.90.31.12:4590 , piece_id=TUKND5MYHBQIH5FKBG6KIFDNAEUAGEX4SZ62D2BJFA77U2KU5UKQ, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.432639544 (320h11m8.984109995s) [noise=true , address=77.90.31.18:4807 , piece_id=QDVLUO3SEXUQJCCMJ4O5VNQK2XBTLXTYKKNZXDFTLXBFRUHMEKEQ, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.435162708 (320h11m8.986633169s) [noise=true , address=145.79.227.15:4874 , piece_id=FE5ZYSKFAKRZZ6RVLVKGZN46PBS5BZFZP5PKVYZNZFBXYGFIKFKA, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.437291813 (320h11m8.988762274s) [noise=true , address=145.79.227.6:4259 , piece_id=JFLTPOWA63S22QGQ6LN4WRFYVV2XQIDRLL7VGWFJRMCPXK5M6MXA, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.439344221 (320h11m8.990814682s) [noise=true , address=77.90.31.19:4841 , piece_id=VKYGMN3H4X2JJUFLNDGZZSCV5HCLQRLR53X5J36FGQ565MKWMNCA, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:10.440379501 (320h11m8.991849973s) [noise=true , address=145.79.227.20:4437 , piece_id=6ION5L5UGDHAJQTYOAC34BTWDUJOJQXDF4XBRF7SQMRY57BYECOQ, seg_pos={0 0}, upload=281977]: Uploading piece... Nov 22 16:15:12.150193113 (320h11m10.701663574s) [upload=281977]: Next returned nil segment Nov 22 16:15:12.150240513 (320h11m10.701710974s) [upload=281977]: Waiting for error group managing segments... Nov 22 16:15:12.562473597 (320h11m11.113944048s) [noise=true , address=77.90.31.15:4699 , piece_id=6Q26DE27DV2TCYPSVMG64R36I4NJFZ6XMSX6ISYYJI6XD3E2TKQA, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:12.579741378 (320h11m11.131211839s) [noise=true , address=77.90.31.12:4590 , piece_id=TUKND5MYHBQIH5FKBG6KIFDNAEUAGEX4SZ62D2BJFA77U2KU5UKQ, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:12.586731719 (320h11m11.13820218s) [noise=true , address=45.91.116.4:4940 , piece_id=G7JVOOU24MGUUAJST54AWGAHJCQ3NEEDNYJCT7274VL3PMXXYEEA, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:12.598257075 (320h11m11.149727536s) [noise=true , address=77.90.31.12:4589 , piece_id=DMJXC2AGYUS6IJDU57YMRVQ4XITS6C65KQPKPK77BAVNIWEDYW6A, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:12.622577225 (320h11m11.174047686s) [noise=true , address=77.90.31.19:4841 , piece_id=VKYGMN3H4X2JJUFLNDGZZSCV5HCLQRLR53X5J36FGQ565MKWMNCA, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:12.754772838 (320h11m11.306243299s) [noise=true , address=77.90.31.18:4807 , piece_id=QDVLUO3SEXUQJCCMJ4O5VNQK2XBTLXTYKKNZXDFTLXBFRUHMEKEQ, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:19.382583530 (320h11m17.934053981s) [noise=true , address=145.79.227.15:4874 , piece_id=FE5ZYSKFAKRZZ6RVLVKGZN46PBS5BZFZP5PKVYZNZFBXYGFIKFKA, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:21.514760656 (320h11m20.066231117s) [noise=true , address=145.79.227.22:4509 , piece_id=XBHW2Q4ZMTXHM3DXV2L3H3UJRHW5NYMLO2Z5WMBPSR3QEKWM4YYA, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:23.636942539 (320h11m22.188413s) [noise=true , address=145.79.227.6:4259 , piece_id=JFLTPOWA63S22QGQ6LN4WRFYVV2XQIDRLL7VGWFJRMCPXK5M6MXA, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:23.872352010 (320h11m22.423822471s) [noise=true , address=145.79.227.28:4725 , piece_id=WAPK2NYLOGLXI4BOM6LA43UNUYKKVGCD6LA2KKPPIMXROR5AHCEA, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:25.966169160 (320h11m24.517639612s) [noise=true , address=145.79.227.8:4331 , piece_id=IXCEZX2MCTDJJWMQY67WUQDLVKUBC6ILSZ66OTDRGZCF43DPPGSQ, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:26.960865746 (320h11m25.512336207s) [noise=true , address=145.79.227.14:4840 , piece_id=IFKFP5EBBMSNGDII2I2MT5IHK223AP2LD2RZLGMI2UFH7EVB6R3A, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:36.322690505 (320h11m34.874160966s) [noise=true , address=145.79.227.21:4474 , piece_id=PECKNOFRUZZIVY5XJ32WCP3CFBJ7LERMD6QFZZPLAXA3B47VNYBQ, seg_pos={0 0}, upload=281977]: Done uploading piece. err: <nil> Nov 22 16:15:36.322718137 (320h11m34.874188599s) [seg_pos={0 0}, upload=281977]: Segment reached optimal threshold of 13 pieces. Nov 22 16:15:36.323447516 (320h11m34.874917977s) [noise=true , address=145.79.227.11:4731 , piece_id=ASYJFIYGHG2VMQE5Y2W6J2MH4HE4TJX5HB36VDPNL6JI6LOBOWMA, seg_pos={0 0}, upload=281977]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9): protocol: rpc: dial tcp 145.79.227.11:4731: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 16:15:36.323459900 (320h11m34.874930371s) [noise=true , address=145.79.227.11:4731 , piece_id=ASYJFIYGHG2VMQE5Y2W6J2MH4HE4TJX5HB36VDPNL6JI6LOBOWMA, seg_pos={0 0}, upload=281977]: Long tail cancellation due to optimal threshold Nov 22 16:15:36.323615366 (320h11m34.875085817s) [noise=true , address=145.79.227.20:4437 , piece_id=6ION5L5UGDHAJQTYOAC34BTWDUJOJQXDF4XBRF7SQMRY57BYECOQ, seg_pos={0 0}, upload=281977]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1G3fe4SHMdNAjAmqLNyFb6qDKL94aAWGYE9qhczKmozKFhKKEV): protocol: write tcp 188.166.225.170:49846->145.79.227.20:4437: use of closed network connection; write tcp 188.166.225.170:49846->145.79.227.20:4437: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49846->145.79.227.20:4437: use of closed network connection Nov 22 16:15:36.323624343 (320h11m34.875094804s) [noise=true , address=145.79.227.20:4437 , piece_id=6ION5L5UGDHAJQTYOAC34BTWDUJOJQXDF4XBRF7SQMRY57BYECOQ, seg_pos={0 0}, upload=281977]: Long tail cancellation due to optimal threshold Nov 22 16:15:36.323867195 (320h11m34.875337677s) [noise=true , address=145.79.227.13:4802 , piece_id=VQ25RRLCCHN2BIEARHDMGO6JXOLM3RAAYSBGDJRZZAA7LU6XIORA, seg_pos={0 0}, upload=281977]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4): protocol: read tcp 188.166.225.170:59064->145.79.227.13:4802: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 16:15:36.323877184 (320h11m34.875347645s) [noise=true , address=145.79.227.13:4802 , piece_id=VQ25RRLCCHN2BIEARHDMGO6JXOLM3RAAYSBGDJRZZAA7LU6XIORA, seg_pos={0 0}, upload=281977]: Long tail cancellation due to optimal threshold Nov 22 16:15:36.323899958 (320h11m34.875370419s) [seg_pos={0 0}, upload=281977]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 22 16:15:36.324029074 (320h11m34.875499535s) [upload=281977]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 22 16:15:36.324036789 (320h11m34.87550726s) [upload=281977]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 22 16:15:36.620020293 (320h11m35.171490744s) [upload=281977]: Done uploading segments... Nov 22 16:15:36.620042956 (320h11m35.171513417s) [upload=281977]: Upload finished. err: <nil> Duration: 14.875802804s core uplink us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)