Orbiter History

checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 39 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.....................] duration: 3.728177739s trace: https://jaeger.dev.storj.io/trace/6af15a3a67b655d5 FAIL (2025-11-21 11:53:41 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 3.728199661s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 2 out of 60 is failed (threshold=3) [................X.X.........................................] duration: 6.687670494s trace: https://jaeger.dev.storj.io/trace/a7f69ee106efb7f FAIL (2025-11-21 11:53:26 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== upload=256586]: Uploading piece... Nov 21 11:25:27.210346024 (291h21m25.761816485s) [noise=true , address=89.149.201.147:4008 , piece_id=HCX7I4ICPEHDWTYDFIQMKZ5EE5CGPQLYAIHF7WZMTUUU6CNZPHTA, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.211425289 (291h21m25.76289575s) [noise=true , address=81.171.14.70:4044 , piece_id=QYUBNGS2FAFHIWMW7GAGYJ3UVIWVP62LIDCOK6QUXEIQNIXRJDAQ, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.213103795 (291h21m25.764574246s) [noise=true , address=185.167.204.205:4022 , piece_id=BES3YK6PYQV2AKOWSAGHF7DGGWGWSWPIJU2ICAQ5WBHKZ5VT5L6A, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.213408395 (291h21m25.764878856s) [noise=true , address=185.167.204.205:4027 , piece_id=IVPONKBF2PIHZQAW3L4YA4HOMQV5BQML6SKKCJLIEGT5SZSQ2AOA, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.214501747 (291h21m25.765972208s) [noise=true , address=81.171.14.70:4042 , piece_id=XNREZEUZ5L5KZ3L7ZOL6IRK7IHU6DJN4RLEQZRFMDKDH2JWC24CQ, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.215013752 (291h21m25.766484213s) [noise=true , address=81.171.14.70:4034 , piece_id=ITRLKYVNFSYBLABNGNFO3HRJNMX3MWB345FIMWSIPBNAVZPA47UA, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.215087572 (291h21m25.766558044s) [noise=true , address=185.167.204.205:4031 , piece_id=ZEI2YSIQC655R7WS43WADXGFT5RKT5LFSN6IZEQ6SHTDVOV7BZBQ, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.216432844 (291h21m25.767903305s) [noise=true , address=185.167.204.206:4056 , piece_id=EAMZIK6AVJL7IMFF5VOLW3SMCGYKSKB2VXJTNUNLJCVIT7EKXNMQ, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.216662281 (291h21m25.768132742s) [noise=true , address=89.149.201.147:4011 , piece_id=ZATNMD5HOK35AWGJJ5V66VDJ4YGUVZQJ2OBERQL32ZV5GW2HBSMQ, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.218008314 (291h21m25.769478775s) [noise=true , address=81.171.14.70:4032 , piece_id=CFH2RTR3BLBUB5ZIZCFC6UHT3FHT3B2XSCHQMKXNGDJJGPHEMCTQ, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:28.035150976 (291h21m26.586621437s) [upload=256586]: Next returned nil segment Nov 21 11:25:28.035169622 (291h21m26.586640083s) [upload=256586]: Waiting for error group managing segments... Nov 21 11:25:28.416646836 (291h21m26.968117317s) [noise=true , address=185.167.204.206:4064 , piece_id=SRFIWFQLRAGCNIKYLHMGT4QRUL3WH2QPLLIKAJ7DLFJEQUW5DPPA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.417338854 (291h21m26.968809325s) [noise=true , address=185.167.204.205:4033 , piece_id=FEWQVHPIKN3Q6O5AZ6AGJBLUPZ34PBZFB5X3XNSKGPOGI4AIO7JA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.421089406 (291h21m26.972559867s) [noise=true , address=185.167.204.206:4072 , piece_id=GYCO53OWGOMOHK4FHC5GENN4LTQGSZKZKOI5V534RLV3LHLPKC6Q, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.426252887 (291h21m26.977723348s) [noise=true , address=185.167.204.206:4046 , piece_id=BZ57ZA35ITEOA6CAZJGNODK5MUTXEZWDV7E2OGBZOSMS4OQZYAEA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.431720689 (291h21m26.98319115s) [noise=true , address=185.167.204.205:4027 , piece_id=IVPONKBF2PIHZQAW3L4YA4HOMQV5BQML6SKKCJLIEGT5SZSQ2AOA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.438243951 (291h21m26.989714412s) [noise=true , address=185.167.204.205:4022 , piece_id=BES3YK6PYQV2AKOWSAGHF7DGGWGWSWPIJU2ICAQ5WBHKZ5VT5L6A, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.443364331 (291h21m26.994834792s) [noise=true , address=185.167.204.206:4056 , piece_id=EAMZIK6AVJL7IMFF5VOLW3SMCGYKSKB2VXJTNUNLJCVIT7EKXNMQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.445594728 (291h21m26.997065179s) [noise=true , address=185.167.204.205:4016 , piece_id=EHQS3546UXRXTL2MGFCSRTPDDPAYXEYLMISPP5AWY337MHX7HFLQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.449325822 (291h21m27.000796283s) [noise=true , address=185.167.204.205:4031 , piece_id=ZEI2YSIQC655R7WS43WADXGFT5RKT5LFSN6IZEQ6SHTDVOV7BZBQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.449761211 (291h21m27.001231672s) [noise=true , address=185.167.204.205:4018 , piece_id=7OSLPWJLKFTIJPZEAFMZOAH6JAAMX6HKROI25FODFYBPNJWSMLWA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.450419845 (291h21m27.001890306s) [noise=true , address=185.167.204.206:4062 , piece_id=UF3S6NAVGS2KQJPGA5WXP7YG4RQO44P445SNPCWXKXMONZAYUTNQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.453201252 (291h21m27.004671713s) [noise=true , address=185.167.204.205:4007 , piece_id=MEBGOPZ6MFMQY4UZARS2LKJKEMNN5YGSLKPJGTWSKS45CUNXSRGQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:29.661209626 (291h21m28.212680087s) [noise=true , address=89.149.201.147:4016 , piece_id=TGRWGGGBKDANC7YKROWPRMIYNAVFEVE3RXOSUOINFI7KTRS2IYUQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:29.693169182 (291h21m28.244639643s) [noise=true , address=89.149.201.147:4002 , piece_id=UIYJSCMY3LCLRHZUJB6JFJHHXKMYTVKERTZW7F7X44MWJETPY6XA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:29.721204497 (291h21m28.272674958s) [noise=true , address=89.149.201.147:4008 , piece_id=HCX7I4ICPEHDWTYDFIQMKZ5EE5CGPQLYAIHF7WZMTUUU6CNZPHTA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:29.764584711 (291h21m28.316055172s) [noise=true , address=89.149.201.147:4001 , piece_id=QUS6NB3NQBLJFB6YCK3EBDPKCTWEWGKAT3NOR24TVI65SQKHWDOQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:29.768284075 (291h21m28.319754536s) [noise=true , address=89.149.201.147:4011 , piece_id=ZATNMD5HOK35AWGJJ5V66VDJ4YGUVZQJ2OBERQL32ZV5GW2HBSMQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:29.789681961 (291h21m28.341152422s) [noise=true , address=89.149.201.147:4003 , piece_id=RUAYGN2RGF5JJTWFEGIHWH3BHEUDSS5DN2QJ6YLSHPF5BGUXD3RA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:37.298598682 (291h21m35.850069143s) [noise=true , address=81.171.14.70:4043 , piece_id=T3NF6JR3VJIGDMPNUBLMJ3AOTQP5IHOGJYORMCSS7ZMOENQETZ7A, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:37.497478356 (291h21m36.048948818s) [noise=true , address=81.171.14.70:4032 , piece_id=CFH2RTR3BLBUB5ZIZCFC6UHT3FHT3B2XSCHQMKXNGDJJGPHEMCTQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:40.406166933 (291h21m38.957637394s) [noise=true , address=81.171.14.70:4045 , piece_id=BRWQ7G3E547PJE2RLGSESIB3HMP5C4M5NSKAJNGWSFXUBEO6SYEQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:42.511041157 (291h21m41.062511608s) [noise=true , address=81.171.14.70:4029 , piece_id=DKJM5E7EY7LOSBSEBQOZRC6CI5ZPPE47HFOFQJROPCUMUV567CAA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:26:12.012682521 (291h22m10.564152982s) [noise=true , address=81.171.14.70:4042 , piece_id=XNREZEUZ5L5KZ3L7ZOL6IRK7IHU6DJN4RLEQZRFMDKDH2JWC24CQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:26:15.596046283 (291h22m14.147516744s) [noise=true , address=81.171.14.70:4033 , piece_id=JRKPD4EP6FZM5HISOIDJYZY7FAWWQCADAS47FGW2JM5A7RFJR2BQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:26:18.453059659 (291h22m17.00453012s) [noise=true , address=81.171.14.70:4046 , piece_id=OSONRSF6KJS7F2QJVCC3FPZK4AV7QPG5YSM47CW5LIW7EME44AWA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:26:23.442583344 (291h22m21.994053805s) [noise=true , address=81.171.14.70:4034 , piece_id=ITRLKYVNFSYBLABNGNFO3HRJNMX3MWB345FIMWSIPBNAVZPA47UA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:26:23.442621967 (291h22m21.994092428s) [seg_pos={0 0}, upload=256586]: Segment reached optimal threshold of 26 pieces. Nov 21 11:26:23.442836976 (291h22m21.994307437s) [noise=true , address=81.171.14.70:4044 , piece_id=QYUBNGS2FAFHIWMW7GAGYJ3UVIWVP62LIDCOK6QUXEIQNIXRJDAQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1MUVGecysnTSFcJoHUL7QMWb3h6dtXgUd7FbA672fQCyEWhePr): protocol: expected piece hash; context canceled; EOF Nov 21 11:26:23.442845934 (291h22m21.994316395s) [noise=true , address=81.171.14.70:4044 , piece_id=QYUBNGS2FAFHIWMW7GAGYJ3UVIWVP62LIDCOK6QUXEIQNIXRJDAQ, seg_pos={0 0}, upload=256586]: Long tail cancellation due to optimal threshold Nov 21 11:26:23.443227491 (291h22m21.994697942s) [noise=true , address=81.171.14.70:4027 , piece_id=4FDFCCWF3EIROQAFLOHGS743523OYGSXWQSPQEVD3AMQ6WKDQI7Q, seg_pos={0 0}, upload=256586]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122cYvgTxSvC2aEXnBkRk2o8ARTBXj9gqPoa12vW8hogSN6kpA4): protocol: expected piece hash; context canceled; EOF Nov 21 11:26:23.443237770 (291h22m21.994708231s) [noise=true , address=81.171.14.70:4027 , piece_id=4FDFCCWF3EIROQAFLOHGS743523OYGSXWQSPQEVD3AMQ6WKDQI7Q, seg_pos={0 0}, upload=256586]: Long tail cancellation due to optimal threshold Nov 21 11:26:23.443274720 (291h22m21.994745181s) [seg_pos={0 0}, upload=256586]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Nov 21 11:26:23.443378929 (291h22m21.99484939s) [upload=256586]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 21 11:26:23.443387044 (291h22m21.994857495s) [upload=256586]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 21 11:26:23.685139626 (291h22m22.236610077s) [upload=256586]: Done uploading segments... Nov 21 11:26:23.685160676 (291h22m22.236631127s) [upload=256586]: Upload finished. err: <nil> Duration: 6.687702463s core uplink eu-select-1 eu1-placement-15
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 40 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX....................] duration: 4.061749811s trace: https://jaeger.dev.storj.io/trace/6af15a3a67b655d5 FAIL (2025-11-21 11:53:13 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 4.061778276s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 16 out of 60 is failed (threshold=3) [.X......X.XX...X.X.X....X............X....X.X.X..X.X..X...X.] duration: 5.781521843s trace: https://jaeger.dev.storj.io/trace/513850953a062c9b FAIL (2025-11-21 11:52:53 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq, Piece ID: QTNPHDNVVEYPLYUQ65RHE6XTRGLTDEIZ3YABFKUGGYZPREECM2RA) context deadline exceeded; piecestore: (Node ID: 123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B, Piece ID: EBJOXQDH4O3QTH5UTFYPKWTRSJPVQWRB3EZOW3QAOZKG5NYGAFQA) context deadline exceeded; piecestore: (Node ID: 12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, Piece ID: ISEEO47BNDEDRTVBYWQNNLRLW3UCKETKU6YOB3U2QBT6765GZMZA) context deadline exceeded ======LOGS====== 21 11:52:18.550541569 (291h48m17.10201203s) [upload=256978]: Starting upload Nov 21 11:52:18.550634156 (291h48m17.102104617s) [upload=256978]: Done starting upload Nov 21 11:52:18.550903869 (291h48m17.10237433s) [upload=256978]: Uploading segments... Nov 21 11:52:18.550955708 (291h48m17.102426169s) [upload=256978]: Got next segment. Inline: false Nov 21 11:52:18.550997727 (291h48m17.102468189s) [upload=256978]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 21 11:52:18.551003969 (291h48m17.10247443s) [upload=256978]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 21 11:52:18.796351348 (291h48m17.347821809s) [seg_pos={0 0}, upload=256978]: Begin upload segment... Nov 21 11:52:18.796415901 (291h48m17.347886352s) [seg_pos={0 0}, upload=256978]: Done begin upload segment. Nov 21 11:52:18.796454224 (291h48m17.347924685s) [noise=true , address=145.79.227.17:4971 , piece_id=SU7OAQ3IQDYLVINT3UNK6M7YDVWGI4DJMXZXXD2AEMKEXPJ4RRFA, seg_pos={0 0}, upload=256978]: Uploading piece... Nov 21 11:52:18.796581256 (291h48m17.348051717s) [noise=true , address=145.79.227.24:4583 , piece_id=YHFQLMSJJ3XMUWK7OJ2247WGMEE3EV5DMPNZNIA5GD7P23K5VITA, seg_pos={0 0}, upload=256978]: Uploading piece... Nov 21 11:52:18.797167943 (291h48m17.348638394s) [noise=true , address=145.79.227.32:4835 , piece_id=VLPQKL7A6II7Q3573BCUIP3YQAMKRTD6I2MKWR55SKTUZ6P2XRHQ, seg_pos={0 0}, upload=256978]: Uploading piece... Nov 21 11:52:18.797249749 (291h48m17.34872021s) [noise=true , address=145.79.227.21:4475 , piece_id=5UJ5JQORO24UGNVTZ4GWHGQIN7Z4YZ2DSQBBJPZTIFWGRAGTPCSA, seg_pos={0 0}, upload=256978]: Uploading piece... Nov 21 11:52:18.797609514 (291h48m17.349079976s) [noise=true , address=145.79.227.30:4762 , piece_id=5A35DXNMNHWWWLA7CLOPJK7VJ4HEPFZ62H6AAQAO3HC5FSQPO7NA, seg_pos={0 0}, upload=256978]: Uploading piece... Nov 21 11:52:18.797681652 (291h48m17.349152113s) [noise=true , address=145.79.227.9:4366 , piece_id=XAP3WCJMA7S4LMDQP6FF5QUXR2C475RKPHXZNWUKFV6WWNFE5B5A, seg_pos={0 0}, upload=256978]: Uploading piece... Nov 21 11:52:18.798011419 (291h48m17.34948187s) [noise=true , address=145.79.227.26:4653 , piece_id=B7UUQOEQLBS4BQGXFT4S5WAADKS7GHIRQ347V74EIEGBFAW2GDQA, seg_pos={0 0}, upload=256978]: Uploading piece... Nov 21 11:52:18.798157647 (291h48m17.349628099s) [noise=true , address=45.91.116.10:4083 , piece_id=EBJOXQDH4O3QTH5UTFYPKWTRSJPVQWRB3EZOW3QAOZKG5NYGAFQA, seg_pos={0 0}, upload=256978]: Uploading piece... Nov 21 11:52:18.798423614 (291h48m17.349894065s) [noise=true , address=77.90.31.12:4592 , piece_id=ISEEO47BNDEDRTVBYWQNNLRLW3UCKETKU6YOB3U2QBT6765GZMZA, seg_pos={0 0}, upload=256978]: Uploading piece... Nov 21 11:52:18.798586044 (291h48m17.350056505s) [noise=true , address=145.79.227.8:4329 , piece_id=OO4DEPELLYK2FTBHKTKGXZKWYURJVDM2XP3ML4TUWC7OC7M3WW5Q, seg_pos={0 0}, upload=256978]: Uploading piece... Nov 21 11:52:18.798812745 (291h48m17.350283196s) [noise=true , address=145.79.227.13:4801 , piece_id=RMIFD5T33USCOVYIBRFHK5T6JRKAPDTESARJBC7AJ3I6AXGQFDWQ, seg_pos={0 0}, upload=256978]: Uploading piece... Nov 21 11:52:18.799017114 (291h48m17.350487575s) [noise=true , address=45.91.116.11:4117 , piece_id=R52CHGV4Z4777BMJ3HUHCJO5COC27VT2LHL34EDD2SYO5CTYNCRA, seg_pos={0 0}, upload=256978]: Uploading piece... Nov 21 11:52:18.799177530 (291h48m17.350647991s) [noise=true , address=145.79.227.10:4402 , piece_id=XTYNEHERZ37JUMFEQ24HCXWY3T4EU3EDID6Q3AHTIDGW6CW4OTSA, seg_pos={0 0}, upload=256978]: Uploading piece... Nov 21 11:52:18.799472952 (291h48m17.350943413s) [noise=true , address=45.91.116.14:4191 , piece_id=QTNPHDNVVEYPLYUQ65RHE6XTRGLTDEIZ3YABFKUGGYZPREECM2RA, seg_pos={0 0}, upload=256978]: Uploading piece... Nov 21 11:52:18.799656962 (291h48m17.351127413s) [noise=true , address=77.90.31.19:4843 , piece_id=GL5JTR6KPVN3MXWTVXXAIT26DCRSVTDDC74XOJBK75Y5XY2P2RFA, seg_pos={0 0}, upload=256978]: Uploading piece... Nov 21 11:52:18.799861031 (291h48m17.351331502s) [noise=true , address=77.90.31.17:4770 , piece_id=ZXV3HX7U6ITP6HA7H2KIEDBBWZBWAIA2XDJ3C3IWXWIU5YGGGA3Q, seg_pos={0 0}, upload=256978]: Uploading piece... Nov 21 11:52:20.405914941 (291h48m18.957385402s) [upload=256978]: Next returned nil segment Nov 21 11:52:20.405946161 (291h48m18.957416622s) [upload=256978]: Waiting for error group managing segments... Nov 21 11:52:20.788637197 (291h48m19.340107658s) [noise=true , address=145.79.227.24:4583 , piece_id=YHFQLMSJJ3XMUWK7OJ2247WGMEE3EV5DMPNZNIA5GD7P23K5VITA, seg_pos={0 0}, upload=256978]: Done uploading piece. err: <nil> Nov 21 11:52:20.792094981 (291h48m19.343565432s) [noise=true , address=145.79.227.21:4475 , piece_id=5UJ5JQORO24UGNVTZ4GWHGQIN7Z4YZ2DSQBBJPZTIFWGRAGTPCSA, seg_pos={0 0}, upload=256978]: Done uploading piece. err: <nil> Nov 21 11:52:20.793185819 (291h48m19.34465628s) [noise=true , address=145.79.227.13:4801 , piece_id=RMIFD5T33USCOVYIBRFHK5T6JRKAPDTESARJBC7AJ3I6AXGQFDWQ, seg_pos={0 0}, upload=256978]: Done uploading piece. err: <nil> Nov 21 11:52:20.795089112 (291h48m19.346559573s) [noise=true , address=145.79.227.9:4366 , piece_id=XAP3WCJMA7S4LMDQP6FF5QUXR2C475RKPHXZNWUKFV6WWNFE5B5A, seg_pos={0 0}, upload=256978]: Done uploading piece. err: <nil> Nov 21 11:52:20.795745082 (291h48m19.347215543s) [noise=true , address=145.79.227.17:4971 , piece_id=SU7OAQ3IQDYLVINT3UNK6M7YDVWGI4DJMXZXXD2AEMKEXPJ4RRFA, seg_pos={0 0}, upload=256978]: Done uploading piece. err: <nil> Nov 21 11:52:20.798623102 (291h48m19.350093563s) [noise=true , address=145.79.227.30:4762 , piece_id=5A35DXNMNHWWWLA7CLOPJK7VJ4HEPFZ62H6AAQAO3HC5FSQPO7NA, seg_pos={0 0}, upload=256978]: Done uploading piece. err: <nil> Nov 21 11:52:20.809441713 (291h48m19.360912174s) [noise=true , address=145.79.227.10:4402 , piece_id=XTYNEHERZ37JUMFEQ24HCXWY3T4EU3EDID6Q3AHTIDGW6CW4OTSA, seg_pos={0 0}, upload=256978]: Done uploading piece. err: <nil> Nov 21 11:52:20.809974137 (291h48m19.361444588s) [noise=true , address=145.79.227.8:4329 , piece_id=OO4DEPELLYK2FTBHKTKGXZKWYURJVDM2XP3ML4TUWC7OC7M3WW5Q, seg_pos={0 0}, upload=256978]: Done uploading piece. err: <nil> Nov 21 11:52:20.813153782 (291h48m19.364624233s) [noise=true , address=145.79.227.32:4835 , piece_id=VLPQKL7A6II7Q3573BCUIP3YQAMKRTD6I2MKWR55SKTUZ6P2XRHQ, seg_pos={0 0}, upload=256978]: Done uploading piece. err: <nil> Nov 21 11:52:20.813743304 (291h48m19.365213765s) [noise=true , address=145.79.227.26:4653 , piece_id=B7UUQOEQLBS4BQGXFT4S5WAADKS7GHIRQ347V74EIEGBFAW2GDQA, seg_pos={0 0}, upload=256978]: Done uploading piece. err: <nil> Nov 21 11:52:21.328226067 (291h48m19.879696538s) [noise=true , address=45.91.116.14:4191 , piece_id=QTNPHDNVVEYPLYUQ65RHE6XTRGLTDEIZ3YABFKUGGYZPREECM2RA, seg_pos={0 0}, upload=256978]: Done uploading piece. err: <nil> Nov 21 11:52:21.416925216 (291h48m19.968395667s) [noise=true , address=45.91.116.10:4083 , piece_id=EBJOXQDH4O3QTH5UTFYPKWTRSJPVQWRB3EZOW3QAOZKG5NYGAFQA, seg_pos={0 0}, upload=256978]: Done uploading piece. err: <nil> Nov 21 11:52:22.476743402 (291h48m21.028213863s) [noise=true , address=77.90.31.12:4592 , piece_id=ISEEO47BNDEDRTVBYWQNNLRLW3UCKETKU6YOB3U2QBT6765GZMZA, seg_pos={0 0}, upload=256978]: Done uploading piece. err: <nil> Nov 21 11:52:22.476768199 (291h48m21.02823866s) [seg_pos={0 0}, upload=256978]: Segment reached optimal threshold of 13 pieces. Nov 21 11:52:22.477024557 (291h48m21.028495018s) [noise=true , address=77.90.31.19:4843 , piece_id=GL5JTR6KPVN3MXWTVXXAIT26DCRSVTDDC74XOJBK75Y5XY2P2RFA, seg_pos={0 0}, upload=256978]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12akCPrMTihqq6CKv7VtDbHhXnWrXygXXhZKgZbftGbymRNzuMb): protocol: expected piece hash; context canceled; EOF Nov 21 11:52:22.477036120 (291h48m21.028506581s) [noise=true , address=77.90.31.19:4843 , piece_id=GL5JTR6KPVN3MXWTVXXAIT26DCRSVTDDC74XOJBK75Y5XY2P2RFA, seg_pos={0 0}, upload=256978]: Long tail cancellation due to optimal threshold Nov 21 11:52:22.477064013 (291h48m21.028534484s) [noise=true , address=77.90.31.17:4770 , piece_id=ZXV3HX7U6ITP6HA7H2KIEDBBWZBWAIA2XDJ3C3IWXWIU5YGGGA3Q, seg_pos={0 0}, upload=256978]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R): protocol: expected piece hash; context canceled; EOF Nov 21 11:52:22.477074463 (291h48m21.028544924s) [noise=true , address=77.90.31.17:4770 , piece_id=ZXV3HX7U6ITP6HA7H2KIEDBBWZBWAIA2XDJ3C3IWXWIU5YGGGA3Q, seg_pos={0 0}, upload=256978]: Long tail cancellation due to optimal threshold Nov 21 11:52:22.477289833 (291h48m21.028760284s) [noise=true , address=45.91.116.11:4117 , piece_id=R52CHGV4Z4777BMJ3HUHCJO5COC27VT2LHL34EDD2SYO5CTYNCRA, seg_pos={0 0}, upload=256978]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6): protocol: write tcp 188.166.225.170:52312->45.91.116.11:4117: use of closed network connection; write tcp 188.166.225.170:52312->45.91.116.11:4117: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52312->45.91.116.11:4117: use of closed network connection Nov 21 11:52:22.477299491 (291h48m21.028769962s) [noise=true , address=45.91.116.11:4117 , piece_id=R52CHGV4Z4777BMJ3HUHCJO5COC27VT2LHL34EDD2SYO5CTYNCRA, seg_pos={0 0}, upload=256978]: Long tail cancellation due to optimal threshold Nov 21 11:52:22.477330400 (291h48m21.028800851s) [seg_pos={0 0}, upload=256978]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 21 11:52:22.477448535 (291h48m21.028918996s) [upload=256978]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 21 11:52:22.477465617 (291h48m21.028936088s) [upload=256978]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 21 11:52:22.771779587 (291h48m21.323250058s) [upload=256978]: Done uploading segments... Nov 21 11:52:22.771808563 (291h48m21.323279014s) [upload=256978]: Upload finished. err: <nil> Duration: 5.781541352s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 17 out of 60 is failed (threshold=3) [X.X......X.XX...X.X.X....X............X....X.X.X..X.X..X...X] duration: 30.002020265s trace: https://jaeger.dev.storj.io/trace/513850953a062c9b FAIL (2025-11-21 11:52:47 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq, Piece ID: QTNPHDNVVEYPLYUQ65RHE6XTRGLTDEIZ3YABFKUGGYZPREECM2RA) context deadline exceeded; piecestore: (Node ID: 123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B, Piece ID: EBJOXQDH4O3QTH5UTFYPKWTRSJPVQWRB3EZOW3QAOZKG5NYGAFQA) context deadline exceeded; piecestore: (Node ID: 12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, Piece ID: ISEEO47BNDEDRTVBYWQNNLRLW3UCKETKU6YOB3U2QBT6765GZMZA) context deadline exceeded ======LOGS====== 21 11:52:18.550541569 (291h48m17.10201203s) [upload=256978]: Starting upload Nov 21 11:52:18.550634156 (291h48m17.102104617s) [upload=256978]: Done starting upload Nov 21 11:52:18.550903869 (291h48m17.10237433s) [upload=256978]: Uploading segments... Nov 21 11:52:18.550955708 (291h48m17.102426169s) [upload=256978]: Got next segment. Inline: false Nov 21 11:52:18.550997727 (291h48m17.102468189s) [upload=256978]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 21 11:52:18.551003969 (291h48m17.10247443s) [upload=256978]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 21 11:52:18.796351348 (291h48m17.347821809s) [seg_pos={0 0}, upload=256978]: Begin upload segment... Nov 21 11:52:18.796415901 (291h48m17.347886352s) [seg_pos={0 0}, upload=256978]: Done begin upload segment. Nov 21 11:52:18.796454224 (291h48m17.347924685s) [noise=true , address=145.79.227.17:4971 , piece_id=SU7OAQ3IQDYLVINT3UNK6M7YDVWGI4DJMXZXXD2AEMKEXPJ4RRFA, seg_pos={0 0}, upload=256978]: Uploading piece... Nov 21 11:52:18.796581256 (291h48m17.348051717s) [noise=true , address=145.79.227.24:4583 , piece_id=YHFQLMSJJ3XMUWK7OJ2247WGMEE3EV5DMPNZNIA5GD7P23K5VITA, seg_pos={0 0}, upload=256978]: Uploading piece... Nov 21 11:52:18.797167943 (291h48m17.348638394s) [noise=true , address=145.79.227.32:4835 , piece_id=VLPQKL7A6II7Q3573BCUIP3YQAMKRTD6I2MKWR55SKTUZ6P2XRHQ, seg_pos={0 0}, upload=256978]: Uploading piece... Nov 21 11:52:18.797249749 (291h48m17.34872021s) [noise=true , address=145.79.227.21:4475 , piece_id=5UJ5JQORO24UGNVTZ4GWHGQIN7Z4YZ2DSQBBJPZTIFWGRAGTPCSA, seg_pos={0 0}, upload=256978]: Uploading piece... Nov 21 11:52:18.797609514 (291h48m17.349079976s) [noise=true , address=145.79.227.30:4762 , piece_id=5A35DXNMNHWWWLA7CLOPJK7VJ4HEPFZ62H6AAQAO3HC5FSQPO7NA, seg_pos={0 0}, upload=256978]: Uploading piece... Nov 21 11:52:18.797681652 (291h48m17.349152113s) [noise=true , address=145.79.227.9:4366 , piece_id=XAP3WCJMA7S4LMDQP6FF5QUXR2C475RKPHXZNWUKFV6WWNFE5B5A, seg_pos={0 0}, upload=256978]: Uploading piece... Nov 21 11:52:18.798011419 (291h48m17.34948187s) [noise=true , address=145.79.227.26:4653 , piece_id=B7UUQOEQLBS4BQGXFT4S5WAADKS7GHIRQ347V74EIEGBFAW2GDQA, seg_pos={0 0}, upload=256978]: Uploading piece... Nov 21 11:52:18.798157647 (291h48m17.349628099s) [noise=true , address=45.91.116.10:4083 , piece_id=EBJOXQDH4O3QTH5UTFYPKWTRSJPVQWRB3EZOW3QAOZKG5NYGAFQA, seg_pos={0 0}, upload=256978]: Uploading piece... Nov 21 11:52:18.798423614 (291h48m17.349894065s) [noise=true , address=77.90.31.12:4592 , piece_id=ISEEO47BNDEDRTVBYWQNNLRLW3UCKETKU6YOB3U2QBT6765GZMZA, seg_pos={0 0}, upload=256978]: Uploading piece... Nov 21 11:52:18.798586044 (291h48m17.350056505s) [noise=true , address=145.79.227.8:4329 , piece_id=OO4DEPELLYK2FTBHKTKGXZKWYURJVDM2XP3ML4TUWC7OC7M3WW5Q, seg_pos={0 0}, upload=256978]: Uploading piece... Nov 21 11:52:18.798812745 (291h48m17.350283196s) [noise=true , address=145.79.227.13:4801 , piece_id=RMIFD5T33USCOVYIBRFHK5T6JRKAPDTESARJBC7AJ3I6AXGQFDWQ, seg_pos={0 0}, upload=256978]: Uploading piece... Nov 21 11:52:18.799017114 (291h48m17.350487575s) [noise=true , address=45.91.116.11:4117 , piece_id=R52CHGV4Z4777BMJ3HUHCJO5COC27VT2LHL34EDD2SYO5CTYNCRA, seg_pos={0 0}, upload=256978]: Uploading piece... Nov 21 11:52:18.799177530 (291h48m17.350647991s) [noise=true , address=145.79.227.10:4402 , piece_id=XTYNEHERZ37JUMFEQ24HCXWY3T4EU3EDID6Q3AHTIDGW6CW4OTSA, seg_pos={0 0}, upload=256978]: Uploading piece... Nov 21 11:52:18.799472952 (291h48m17.350943413s) [noise=true , address=45.91.116.14:4191 , piece_id=QTNPHDNVVEYPLYUQ65RHE6XTRGLTDEIZ3YABFKUGGYZPREECM2RA, seg_pos={0 0}, upload=256978]: Uploading piece... Nov 21 11:52:18.799656962 (291h48m17.351127413s) [noise=true , address=77.90.31.19:4843 , piece_id=GL5JTR6KPVN3MXWTVXXAIT26DCRSVTDDC74XOJBK75Y5XY2P2RFA, seg_pos={0 0}, upload=256978]: Uploading piece... Nov 21 11:52:18.799861031 (291h48m17.351331502s) [noise=true , address=77.90.31.17:4770 , piece_id=ZXV3HX7U6ITP6HA7H2KIEDBBWZBWAIA2XDJ3C3IWXWIU5YGGGA3Q, seg_pos={0 0}, upload=256978]: Uploading piece... Nov 21 11:52:20.405914941 (291h48m18.957385402s) [upload=256978]: Next returned nil segment Nov 21 11:52:20.405946161 (291h48m18.957416622s) [upload=256978]: Waiting for error group managing segments... Nov 21 11:52:20.788637197 (291h48m19.340107658s) [noise=true , address=145.79.227.24:4583 , piece_id=YHFQLMSJJ3XMUWK7OJ2247WGMEE3EV5DMPNZNIA5GD7P23K5VITA, seg_pos={0 0}, upload=256978]: Done uploading piece. err: <nil> Nov 21 11:52:20.792094981 (291h48m19.343565432s) [noise=true , address=145.79.227.21:4475 , piece_id=5UJ5JQORO24UGNVTZ4GWHGQIN7Z4YZ2DSQBBJPZTIFWGRAGTPCSA, seg_pos={0 0}, upload=256978]: Done uploading piece. err: <nil> Nov 21 11:52:20.793185819 (291h48m19.34465628s) [noise=true , address=145.79.227.13:4801 , piece_id=RMIFD5T33USCOVYIBRFHK5T6JRKAPDTESARJBC7AJ3I6AXGQFDWQ, seg_pos={0 0}, upload=256978]: Done uploading piece. err: <nil> Nov 21 11:52:20.795089112 (291h48m19.346559573s) [noise=true , address=145.79.227.9:4366 , piece_id=XAP3WCJMA7S4LMDQP6FF5QUXR2C475RKPHXZNWUKFV6WWNFE5B5A, seg_pos={0 0}, upload=256978]: Done uploading piece. err: <nil> Nov 21 11:52:20.795745082 (291h48m19.347215543s) [noise=true , address=145.79.227.17:4971 , piece_id=SU7OAQ3IQDYLVINT3UNK6M7YDVWGI4DJMXZXXD2AEMKEXPJ4RRFA, seg_pos={0 0}, upload=256978]: Done uploading piece. err: <nil> Nov 21 11:52:20.798623102 (291h48m19.350093563s) [noise=true , address=145.79.227.30:4762 , piece_id=5A35DXNMNHWWWLA7CLOPJK7VJ4HEPFZ62H6AAQAO3HC5FSQPO7NA, seg_pos={0 0}, upload=256978]: Done uploading piece. err: <nil> Nov 21 11:52:20.809441713 (291h48m19.360912174s) [noise=true , address=145.79.227.10:4402 , piece_id=XTYNEHERZ37JUMFEQ24HCXWY3T4EU3EDID6Q3AHTIDGW6CW4OTSA, seg_pos={0 0}, upload=256978]: Done uploading piece. err: <nil> Nov 21 11:52:20.809974137 (291h48m19.361444588s) [noise=true , address=145.79.227.8:4329 , piece_id=OO4DEPELLYK2FTBHKTKGXZKWYURJVDM2XP3ML4TUWC7OC7M3WW5Q, seg_pos={0 0}, upload=256978]: Done uploading piece. err: <nil> Nov 21 11:52:20.813153782 (291h48m19.364624233s) [noise=true , address=145.79.227.32:4835 , piece_id=VLPQKL7A6II7Q3573BCUIP3YQAMKRTD6I2MKWR55SKTUZ6P2XRHQ, seg_pos={0 0}, upload=256978]: Done uploading piece. err: <nil> Nov 21 11:52:20.813743304 (291h48m19.365213765s) [noise=true , address=145.79.227.26:4653 , piece_id=B7UUQOEQLBS4BQGXFT4S5WAADKS7GHIRQ347V74EIEGBFAW2GDQA, seg_pos={0 0}, upload=256978]: Done uploading piece. err: <nil> Nov 21 11:52:21.328226067 (291h48m19.879696538s) [noise=true , address=45.91.116.14:4191 , piece_id=QTNPHDNVVEYPLYUQ65RHE6XTRGLTDEIZ3YABFKUGGYZPREECM2RA, seg_pos={0 0}, upload=256978]: Done uploading piece. err: <nil> Nov 21 11:52:21.416925216 (291h48m19.968395667s) [noise=true , address=45.91.116.10:4083 , piece_id=EBJOXQDH4O3QTH5UTFYPKWTRSJPVQWRB3EZOW3QAOZKG5NYGAFQA, seg_pos={0 0}, upload=256978]: Done uploading piece. err: <nil> Nov 21 11:52:22.476743402 (291h48m21.028213863s) [noise=true , address=77.90.31.12:4592 , piece_id=ISEEO47BNDEDRTVBYWQNNLRLW3UCKETKU6YOB3U2QBT6765GZMZA, seg_pos={0 0}, upload=256978]: Done uploading piece. err: <nil> Nov 21 11:52:22.476768199 (291h48m21.02823866s) [seg_pos={0 0}, upload=256978]: Segment reached optimal threshold of 13 pieces. Nov 21 11:52:22.477024557 (291h48m21.028495018s) [noise=true , address=77.90.31.19:4843 , piece_id=GL5JTR6KPVN3MXWTVXXAIT26DCRSVTDDC74XOJBK75Y5XY2P2RFA, seg_pos={0 0}, upload=256978]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12akCPrMTihqq6CKv7VtDbHhXnWrXygXXhZKgZbftGbymRNzuMb): protocol: expected piece hash; context canceled; EOF Nov 21 11:52:22.477036120 (291h48m21.028506581s) [noise=true , address=77.90.31.19:4843 , piece_id=GL5JTR6KPVN3MXWTVXXAIT26DCRSVTDDC74XOJBK75Y5XY2P2RFA, seg_pos={0 0}, upload=256978]: Long tail cancellation due to optimal threshold Nov 21 11:52:22.477064013 (291h48m21.028534484s) [noise=true , address=77.90.31.17:4770 , piece_id=ZXV3HX7U6ITP6HA7H2KIEDBBWZBWAIA2XDJ3C3IWXWIU5YGGGA3Q, seg_pos={0 0}, upload=256978]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R): protocol: expected piece hash; context canceled; EOF Nov 21 11:52:22.477074463 (291h48m21.028544924s) [noise=true , address=77.90.31.17:4770 , piece_id=ZXV3HX7U6ITP6HA7H2KIEDBBWZBWAIA2XDJ3C3IWXWIU5YGGGA3Q, seg_pos={0 0}, upload=256978]: Long tail cancellation due to optimal threshold Nov 21 11:52:22.477289833 (291h48m21.028760284s) [noise=true , address=45.91.116.11:4117 , piece_id=R52CHGV4Z4777BMJ3HUHCJO5COC27VT2LHL34EDD2SYO5CTYNCRA, seg_pos={0 0}, upload=256978]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6): protocol: write tcp 188.166.225.170:52312->45.91.116.11:4117: use of closed network connection; write tcp 188.166.225.170:52312->45.91.116.11:4117: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52312->45.91.116.11:4117: use of closed network connection Nov 21 11:52:22.477299491 (291h48m21.028769962s) [noise=true , address=45.91.116.11:4117 , piece_id=R52CHGV4Z4777BMJ3HUHCJO5COC27VT2LHL34EDD2SYO5CTYNCRA, seg_pos={0 0}, upload=256978]: Long tail cancellation due to optimal threshold Nov 21 11:52:22.477330400 (291h48m21.028800851s) [seg_pos={0 0}, upload=256978]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 21 11:52:22.477448535 (291h48m21.028918996s) [upload=256978]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 21 11:52:22.477465617 (291h48m21.028936088s) [upload=256978]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 21 11:52:22.771779587 (291h48m21.323250058s) [upload=256978]: Done uploading segments... Nov 21 11:52:22.771808563 (291h48m21.323279014s) [upload=256978]: Upload finished. err: <nil> Duration: 30.002047107s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 41 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX...................] duration: 3.653921829s trace: https://jaeger.dev.storj.io/trace/6af15a3a67b655d5 FAIL (2025-11-21 11:52:21 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 3.653961384s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 2 out of 60 is failed (threshold=3) [.................X.X........................................] duration: 5.308869665s trace: https://jaeger.dev.storj.io/trace/a7f69ee106efb7f FAIL (2025-11-21 11:52:18 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== upload=256586]: Uploading piece... Nov 21 11:25:27.210346024 (291h21m25.761816485s) [noise=true , address=89.149.201.147:4008 , piece_id=HCX7I4ICPEHDWTYDFIQMKZ5EE5CGPQLYAIHF7WZMTUUU6CNZPHTA, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.211425289 (291h21m25.76289575s) [noise=true , address=81.171.14.70:4044 , piece_id=QYUBNGS2FAFHIWMW7GAGYJ3UVIWVP62LIDCOK6QUXEIQNIXRJDAQ, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.213103795 (291h21m25.764574246s) [noise=true , address=185.167.204.205:4022 , piece_id=BES3YK6PYQV2AKOWSAGHF7DGGWGWSWPIJU2ICAQ5WBHKZ5VT5L6A, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.213408395 (291h21m25.764878856s) [noise=true , address=185.167.204.205:4027 , piece_id=IVPONKBF2PIHZQAW3L4YA4HOMQV5BQML6SKKCJLIEGT5SZSQ2AOA, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.214501747 (291h21m25.765972208s) [noise=true , address=81.171.14.70:4042 , piece_id=XNREZEUZ5L5KZ3L7ZOL6IRK7IHU6DJN4RLEQZRFMDKDH2JWC24CQ, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.215013752 (291h21m25.766484213s) [noise=true , address=81.171.14.70:4034 , piece_id=ITRLKYVNFSYBLABNGNFO3HRJNMX3MWB345FIMWSIPBNAVZPA47UA, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.215087572 (291h21m25.766558044s) [noise=true , address=185.167.204.205:4031 , piece_id=ZEI2YSIQC655R7WS43WADXGFT5RKT5LFSN6IZEQ6SHTDVOV7BZBQ, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.216432844 (291h21m25.767903305s) [noise=true , address=185.167.204.206:4056 , piece_id=EAMZIK6AVJL7IMFF5VOLW3SMCGYKSKB2VXJTNUNLJCVIT7EKXNMQ, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.216662281 (291h21m25.768132742s) [noise=true , address=89.149.201.147:4011 , piece_id=ZATNMD5HOK35AWGJJ5V66VDJ4YGUVZQJ2OBERQL32ZV5GW2HBSMQ, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.218008314 (291h21m25.769478775s) [noise=true , address=81.171.14.70:4032 , piece_id=CFH2RTR3BLBUB5ZIZCFC6UHT3FHT3B2XSCHQMKXNGDJJGPHEMCTQ, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:28.035150976 (291h21m26.586621437s) [upload=256586]: Next returned nil segment Nov 21 11:25:28.035169622 (291h21m26.586640083s) [upload=256586]: Waiting for error group managing segments... Nov 21 11:25:28.416646836 (291h21m26.968117317s) [noise=true , address=185.167.204.206:4064 , piece_id=SRFIWFQLRAGCNIKYLHMGT4QRUL3WH2QPLLIKAJ7DLFJEQUW5DPPA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.417338854 (291h21m26.968809325s) [noise=true , address=185.167.204.205:4033 , piece_id=FEWQVHPIKN3Q6O5AZ6AGJBLUPZ34PBZFB5X3XNSKGPOGI4AIO7JA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.421089406 (291h21m26.972559867s) [noise=true , address=185.167.204.206:4072 , piece_id=GYCO53OWGOMOHK4FHC5GENN4LTQGSZKZKOI5V534RLV3LHLPKC6Q, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.426252887 (291h21m26.977723348s) [noise=true , address=185.167.204.206:4046 , piece_id=BZ57ZA35ITEOA6CAZJGNODK5MUTXEZWDV7E2OGBZOSMS4OQZYAEA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.431720689 (291h21m26.98319115s) [noise=true , address=185.167.204.205:4027 , piece_id=IVPONKBF2PIHZQAW3L4YA4HOMQV5BQML6SKKCJLIEGT5SZSQ2AOA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.438243951 (291h21m26.989714412s) [noise=true , address=185.167.204.205:4022 , piece_id=BES3YK6PYQV2AKOWSAGHF7DGGWGWSWPIJU2ICAQ5WBHKZ5VT5L6A, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.443364331 (291h21m26.994834792s) [noise=true , address=185.167.204.206:4056 , piece_id=EAMZIK6AVJL7IMFF5VOLW3SMCGYKSKB2VXJTNUNLJCVIT7EKXNMQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.445594728 (291h21m26.997065179s) [noise=true , address=185.167.204.205:4016 , piece_id=EHQS3546UXRXTL2MGFCSRTPDDPAYXEYLMISPP5AWY337MHX7HFLQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.449325822 (291h21m27.000796283s) [noise=true , address=185.167.204.205:4031 , piece_id=ZEI2YSIQC655R7WS43WADXGFT5RKT5LFSN6IZEQ6SHTDVOV7BZBQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.449761211 (291h21m27.001231672s) [noise=true , address=185.167.204.205:4018 , piece_id=7OSLPWJLKFTIJPZEAFMZOAH6JAAMX6HKROI25FODFYBPNJWSMLWA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.450419845 (291h21m27.001890306s) [noise=true , address=185.167.204.206:4062 , piece_id=UF3S6NAVGS2KQJPGA5WXP7YG4RQO44P445SNPCWXKXMONZAYUTNQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.453201252 (291h21m27.004671713s) [noise=true , address=185.167.204.205:4007 , piece_id=MEBGOPZ6MFMQY4UZARS2LKJKEMNN5YGSLKPJGTWSKS45CUNXSRGQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:29.661209626 (291h21m28.212680087s) [noise=true , address=89.149.201.147:4016 , piece_id=TGRWGGGBKDANC7YKROWPRMIYNAVFEVE3RXOSUOINFI7KTRS2IYUQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:29.693169182 (291h21m28.244639643s) [noise=true , address=89.149.201.147:4002 , piece_id=UIYJSCMY3LCLRHZUJB6JFJHHXKMYTVKERTZW7F7X44MWJETPY6XA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:29.721204497 (291h21m28.272674958s) [noise=true , address=89.149.201.147:4008 , piece_id=HCX7I4ICPEHDWTYDFIQMKZ5EE5CGPQLYAIHF7WZMTUUU6CNZPHTA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:29.764584711 (291h21m28.316055172s) [noise=true , address=89.149.201.147:4001 , piece_id=QUS6NB3NQBLJFB6YCK3EBDPKCTWEWGKAT3NOR24TVI65SQKHWDOQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:29.768284075 (291h21m28.319754536s) [noise=true , address=89.149.201.147:4011 , piece_id=ZATNMD5HOK35AWGJJ5V66VDJ4YGUVZQJ2OBERQL32ZV5GW2HBSMQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:29.789681961 (291h21m28.341152422s) [noise=true , address=89.149.201.147:4003 , piece_id=RUAYGN2RGF5JJTWFEGIHWH3BHEUDSS5DN2QJ6YLSHPF5BGUXD3RA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:37.298598682 (291h21m35.850069143s) [noise=true , address=81.171.14.70:4043 , piece_id=T3NF6JR3VJIGDMPNUBLMJ3AOTQP5IHOGJYORMCSS7ZMOENQETZ7A, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:37.497478356 (291h21m36.048948818s) [noise=true , address=81.171.14.70:4032 , piece_id=CFH2RTR3BLBUB5ZIZCFC6UHT3FHT3B2XSCHQMKXNGDJJGPHEMCTQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:40.406166933 (291h21m38.957637394s) [noise=true , address=81.171.14.70:4045 , piece_id=BRWQ7G3E547PJE2RLGSESIB3HMP5C4M5NSKAJNGWSFXUBEO6SYEQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:42.511041157 (291h21m41.062511608s) [noise=true , address=81.171.14.70:4029 , piece_id=DKJM5E7EY7LOSBSEBQOZRC6CI5ZPPE47HFOFQJROPCUMUV567CAA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:26:12.012682521 (291h22m10.564152982s) [noise=true , address=81.171.14.70:4042 , piece_id=XNREZEUZ5L5KZ3L7ZOL6IRK7IHU6DJN4RLEQZRFMDKDH2JWC24CQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:26:15.596046283 (291h22m14.147516744s) [noise=true , address=81.171.14.70:4033 , piece_id=JRKPD4EP6FZM5HISOIDJYZY7FAWWQCADAS47FGW2JM5A7RFJR2BQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:26:18.453059659 (291h22m17.00453012s) [noise=true , address=81.171.14.70:4046 , piece_id=OSONRSF6KJS7F2QJVCC3FPZK4AV7QPG5YSM47CW5LIW7EME44AWA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:26:23.442583344 (291h22m21.994053805s) [noise=true , address=81.171.14.70:4034 , piece_id=ITRLKYVNFSYBLABNGNFO3HRJNMX3MWB345FIMWSIPBNAVZPA47UA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:26:23.442621967 (291h22m21.994092428s) [seg_pos={0 0}, upload=256586]: Segment reached optimal threshold of 26 pieces. Nov 21 11:26:23.442836976 (291h22m21.994307437s) [noise=true , address=81.171.14.70:4044 , piece_id=QYUBNGS2FAFHIWMW7GAGYJ3UVIWVP62LIDCOK6QUXEIQNIXRJDAQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1MUVGecysnTSFcJoHUL7QMWb3h6dtXgUd7FbA672fQCyEWhePr): protocol: expected piece hash; context canceled; EOF Nov 21 11:26:23.442845934 (291h22m21.994316395s) [noise=true , address=81.171.14.70:4044 , piece_id=QYUBNGS2FAFHIWMW7GAGYJ3UVIWVP62LIDCOK6QUXEIQNIXRJDAQ, seg_pos={0 0}, upload=256586]: Long tail cancellation due to optimal threshold Nov 21 11:26:23.443227491 (291h22m21.994697942s) [noise=true , address=81.171.14.70:4027 , piece_id=4FDFCCWF3EIROQAFLOHGS743523OYGSXWQSPQEVD3AMQ6WKDQI7Q, seg_pos={0 0}, upload=256586]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122cYvgTxSvC2aEXnBkRk2o8ARTBXj9gqPoa12vW8hogSN6kpA4): protocol: expected piece hash; context canceled; EOF Nov 21 11:26:23.443237770 (291h22m21.994708231s) [noise=true , address=81.171.14.70:4027 , piece_id=4FDFCCWF3EIROQAFLOHGS743523OYGSXWQSPQEVD3AMQ6WKDQI7Q, seg_pos={0 0}, upload=256586]: Long tail cancellation due to optimal threshold Nov 21 11:26:23.443274720 (291h22m21.994745181s) [seg_pos={0 0}, upload=256586]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Nov 21 11:26:23.443378929 (291h22m21.99484939s) [upload=256586]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 21 11:26:23.443387044 (291h22m21.994857495s) [upload=256586]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 21 11:26:23.685139626 (291h22m22.236610077s) [upload=256586]: Done uploading segments... Nov 21 11:26:23.685160676 (291h22m22.236631127s) [upload=256586]: Upload finished. err: <nil> Duration: 5.308895965s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 2 out of 60 is failed (threshold=3) [..................X.X.......................................] duration: 6.00046312s trace: https://jaeger.dev.storj.io/trace/a7f69ee106efb7f FAIL (2025-11-21 11:52:01 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== upload=256586]: Uploading piece... Nov 21 11:25:27.210346024 (291h21m25.761816485s) [noise=true , address=89.149.201.147:4008 , piece_id=HCX7I4ICPEHDWTYDFIQMKZ5EE5CGPQLYAIHF7WZMTUUU6CNZPHTA, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.211425289 (291h21m25.76289575s) [noise=true , address=81.171.14.70:4044 , piece_id=QYUBNGS2FAFHIWMW7GAGYJ3UVIWVP62LIDCOK6QUXEIQNIXRJDAQ, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.213103795 (291h21m25.764574246s) [noise=true , address=185.167.204.205:4022 , piece_id=BES3YK6PYQV2AKOWSAGHF7DGGWGWSWPIJU2ICAQ5WBHKZ5VT5L6A, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.213408395 (291h21m25.764878856s) [noise=true , address=185.167.204.205:4027 , piece_id=IVPONKBF2PIHZQAW3L4YA4HOMQV5BQML6SKKCJLIEGT5SZSQ2AOA, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.214501747 (291h21m25.765972208s) [noise=true , address=81.171.14.70:4042 , piece_id=XNREZEUZ5L5KZ3L7ZOL6IRK7IHU6DJN4RLEQZRFMDKDH2JWC24CQ, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.215013752 (291h21m25.766484213s) [noise=true , address=81.171.14.70:4034 , piece_id=ITRLKYVNFSYBLABNGNFO3HRJNMX3MWB345FIMWSIPBNAVZPA47UA, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.215087572 (291h21m25.766558044s) [noise=true , address=185.167.204.205:4031 , piece_id=ZEI2YSIQC655R7WS43WADXGFT5RKT5LFSN6IZEQ6SHTDVOV7BZBQ, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.216432844 (291h21m25.767903305s) [noise=true , address=185.167.204.206:4056 , piece_id=EAMZIK6AVJL7IMFF5VOLW3SMCGYKSKB2VXJTNUNLJCVIT7EKXNMQ, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.216662281 (291h21m25.768132742s) [noise=true , address=89.149.201.147:4011 , piece_id=ZATNMD5HOK35AWGJJ5V66VDJ4YGUVZQJ2OBERQL32ZV5GW2HBSMQ, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.218008314 (291h21m25.769478775s) [noise=true , address=81.171.14.70:4032 , piece_id=CFH2RTR3BLBUB5ZIZCFC6UHT3FHT3B2XSCHQMKXNGDJJGPHEMCTQ, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:28.035150976 (291h21m26.586621437s) [upload=256586]: Next returned nil segment Nov 21 11:25:28.035169622 (291h21m26.586640083s) [upload=256586]: Waiting for error group managing segments... Nov 21 11:25:28.416646836 (291h21m26.968117317s) [noise=true , address=185.167.204.206:4064 , piece_id=SRFIWFQLRAGCNIKYLHMGT4QRUL3WH2QPLLIKAJ7DLFJEQUW5DPPA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.417338854 (291h21m26.968809325s) [noise=true , address=185.167.204.205:4033 , piece_id=FEWQVHPIKN3Q6O5AZ6AGJBLUPZ34PBZFB5X3XNSKGPOGI4AIO7JA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.421089406 (291h21m26.972559867s) [noise=true , address=185.167.204.206:4072 , piece_id=GYCO53OWGOMOHK4FHC5GENN4LTQGSZKZKOI5V534RLV3LHLPKC6Q, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.426252887 (291h21m26.977723348s) [noise=true , address=185.167.204.206:4046 , piece_id=BZ57ZA35ITEOA6CAZJGNODK5MUTXEZWDV7E2OGBZOSMS4OQZYAEA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.431720689 (291h21m26.98319115s) [noise=true , address=185.167.204.205:4027 , piece_id=IVPONKBF2PIHZQAW3L4YA4HOMQV5BQML6SKKCJLIEGT5SZSQ2AOA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.438243951 (291h21m26.989714412s) [noise=true , address=185.167.204.205:4022 , piece_id=BES3YK6PYQV2AKOWSAGHF7DGGWGWSWPIJU2ICAQ5WBHKZ5VT5L6A, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.443364331 (291h21m26.994834792s) [noise=true , address=185.167.204.206:4056 , piece_id=EAMZIK6AVJL7IMFF5VOLW3SMCGYKSKB2VXJTNUNLJCVIT7EKXNMQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.445594728 (291h21m26.997065179s) [noise=true , address=185.167.204.205:4016 , piece_id=EHQS3546UXRXTL2MGFCSRTPDDPAYXEYLMISPP5AWY337MHX7HFLQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.449325822 (291h21m27.000796283s) [noise=true , address=185.167.204.205:4031 , piece_id=ZEI2YSIQC655R7WS43WADXGFT5RKT5LFSN6IZEQ6SHTDVOV7BZBQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.449761211 (291h21m27.001231672s) [noise=true , address=185.167.204.205:4018 , piece_id=7OSLPWJLKFTIJPZEAFMZOAH6JAAMX6HKROI25FODFYBPNJWSMLWA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.450419845 (291h21m27.001890306s) [noise=true , address=185.167.204.206:4062 , piece_id=UF3S6NAVGS2KQJPGA5WXP7YG4RQO44P445SNPCWXKXMONZAYUTNQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.453201252 (291h21m27.004671713s) [noise=true , address=185.167.204.205:4007 , piece_id=MEBGOPZ6MFMQY4UZARS2LKJKEMNN5YGSLKPJGTWSKS45CUNXSRGQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:29.661209626 (291h21m28.212680087s) [noise=true , address=89.149.201.147:4016 , piece_id=TGRWGGGBKDANC7YKROWPRMIYNAVFEVE3RXOSUOINFI7KTRS2IYUQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:29.693169182 (291h21m28.244639643s) [noise=true , address=89.149.201.147:4002 , piece_id=UIYJSCMY3LCLRHZUJB6JFJHHXKMYTVKERTZW7F7X44MWJETPY6XA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:29.721204497 (291h21m28.272674958s) [noise=true , address=89.149.201.147:4008 , piece_id=HCX7I4ICPEHDWTYDFIQMKZ5EE5CGPQLYAIHF7WZMTUUU6CNZPHTA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:29.764584711 (291h21m28.316055172s) [noise=true , address=89.149.201.147:4001 , piece_id=QUS6NB3NQBLJFB6YCK3EBDPKCTWEWGKAT3NOR24TVI65SQKHWDOQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:29.768284075 (291h21m28.319754536s) [noise=true , address=89.149.201.147:4011 , piece_id=ZATNMD5HOK35AWGJJ5V66VDJ4YGUVZQJ2OBERQL32ZV5GW2HBSMQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:29.789681961 (291h21m28.341152422s) [noise=true , address=89.149.201.147:4003 , piece_id=RUAYGN2RGF5JJTWFEGIHWH3BHEUDSS5DN2QJ6YLSHPF5BGUXD3RA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:37.298598682 (291h21m35.850069143s) [noise=true , address=81.171.14.70:4043 , piece_id=T3NF6JR3VJIGDMPNUBLMJ3AOTQP5IHOGJYORMCSS7ZMOENQETZ7A, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:37.497478356 (291h21m36.048948818s) [noise=true , address=81.171.14.70:4032 , piece_id=CFH2RTR3BLBUB5ZIZCFC6UHT3FHT3B2XSCHQMKXNGDJJGPHEMCTQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:40.406166933 (291h21m38.957637394s) [noise=true , address=81.171.14.70:4045 , piece_id=BRWQ7G3E547PJE2RLGSESIB3HMP5C4M5NSKAJNGWSFXUBEO6SYEQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:42.511041157 (291h21m41.062511608s) [noise=true , address=81.171.14.70:4029 , piece_id=DKJM5E7EY7LOSBSEBQOZRC6CI5ZPPE47HFOFQJROPCUMUV567CAA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:26:12.012682521 (291h22m10.564152982s) [noise=true , address=81.171.14.70:4042 , piece_id=XNREZEUZ5L5KZ3L7ZOL6IRK7IHU6DJN4RLEQZRFMDKDH2JWC24CQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:26:15.596046283 (291h22m14.147516744s) [noise=true , address=81.171.14.70:4033 , piece_id=JRKPD4EP6FZM5HISOIDJYZY7FAWWQCADAS47FGW2JM5A7RFJR2BQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:26:18.453059659 (291h22m17.00453012s) [noise=true , address=81.171.14.70:4046 , piece_id=OSONRSF6KJS7F2QJVCC3FPZK4AV7QPG5YSM47CW5LIW7EME44AWA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:26:23.442583344 (291h22m21.994053805s) [noise=true , address=81.171.14.70:4034 , piece_id=ITRLKYVNFSYBLABNGNFO3HRJNMX3MWB345FIMWSIPBNAVZPA47UA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:26:23.442621967 (291h22m21.994092428s) [seg_pos={0 0}, upload=256586]: Segment reached optimal threshold of 26 pieces. Nov 21 11:26:23.442836976 (291h22m21.994307437s) [noise=true , address=81.171.14.70:4044 , piece_id=QYUBNGS2FAFHIWMW7GAGYJ3UVIWVP62LIDCOK6QUXEIQNIXRJDAQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1MUVGecysnTSFcJoHUL7QMWb3h6dtXgUd7FbA672fQCyEWhePr): protocol: expected piece hash; context canceled; EOF Nov 21 11:26:23.442845934 (291h22m21.994316395s) [noise=true , address=81.171.14.70:4044 , piece_id=QYUBNGS2FAFHIWMW7GAGYJ3UVIWVP62LIDCOK6QUXEIQNIXRJDAQ, seg_pos={0 0}, upload=256586]: Long tail cancellation due to optimal threshold Nov 21 11:26:23.443227491 (291h22m21.994697942s) [noise=true , address=81.171.14.70:4027 , piece_id=4FDFCCWF3EIROQAFLOHGS743523OYGSXWQSPQEVD3AMQ6WKDQI7Q, seg_pos={0 0}, upload=256586]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122cYvgTxSvC2aEXnBkRk2o8ARTBXj9gqPoa12vW8hogSN6kpA4): protocol: expected piece hash; context canceled; EOF Nov 21 11:26:23.443237770 (291h22m21.994708231s) [noise=true , address=81.171.14.70:4027 , piece_id=4FDFCCWF3EIROQAFLOHGS743523OYGSXWQSPQEVD3AMQ6WKDQI7Q, seg_pos={0 0}, upload=256586]: Long tail cancellation due to optimal threshold Nov 21 11:26:23.443274720 (291h22m21.994745181s) [seg_pos={0 0}, upload=256586]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Nov 21 11:26:23.443378929 (291h22m21.99484939s) [upload=256586]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 21 11:26:23.443387044 (291h22m21.994857495s) [upload=256586]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 21 11:26:23.685139626 (291h22m22.236610077s) [upload=256586]: Done uploading segments... Nov 21 11:26:23.685160676 (291h22m22.236631127s) [upload=256586]: Upload finished. err: <nil> Duration: 6.000496994s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 16 out of 60 is failed (threshold=3) [.X.X......X.XX...X.X.X....X............X....X.X.X..X.X..X...] duration: 6.343416669s trace: https://jaeger.dev.storj.io/trace/34d009b2527f9379 FAIL (2025-11-21 11:51:57 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB, Piece ID: N7NO3DOXYOQDEAGQ43OPWR3E5CBYZX5J4A4EJV2VLGKOKJBXCIQQ) context deadline exceeded; piecestore: (Node ID: 1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ, Piece ID: HLV5EPKQNPRWQKHNJLB5ATKRWSS6XW3HMWYXUJL4L7VM7VL4MBZQ) context deadline exceeded; piecestore: (Node ID: 1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, Piece ID: T3KCZD5JV6FAEG6VHBK6HG2E2C6XMP4A6NVAWFWKZTWJIA57L4AA) context deadline exceeded; piecestore: (Node ID: 155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, Piece ID: CECM54T2XTGYW4BFN4TZY2DYUH47J6YO6PGF2QQ6SOCUMLSW6XMQ) context deadline exceeded ======LOGS====== ments... Nov 21 11:49:16.994612805 (291h45m15.546083266s) [upload=256935]: Got next segment. Inline: false Nov 21 11:49:16.994647011 (291h45m15.546117472s) [upload=256935]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 21 11:49:16.994651759 (291h45m15.54612221s) [upload=256935]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 21 11:49:17.243148153 (291h45m15.794618604s) [seg_pos={0 0}, upload=256935]: Begin upload segment... Nov 21 11:49:17.243197467 (291h45m15.794667929s) [seg_pos={0 0}, upload=256935]: Done begin upload segment. Nov 21 11:49:17.243223908 (291h45m15.794694369s) [noise=true , address=77.90.31.11:4555 , piece_id=HLV5EPKQNPRWQKHNJLB5ATKRWSS6XW3HMWYXUJL4L7VM7VL4MBZQ, seg_pos={0 0}, upload=256935]: Uploading piece... Nov 21 11:49:17.243288660 (291h45m15.794759112s) [noise=true , address=145.79.227.11:4731 , piece_id=2NO5DRKFPM22XIWMQU4GDYHKXWYQX2QRAYP3YQBSIWTLAMW6LAPA, seg_pos={0 0}, upload=256935]: Uploading piece... Nov 21 11:49:17.243999364 (291h45m15.795469815s) [noise=true , address=45.91.116.14:4191 , piece_id=3TMBBNLJQ4OZ7RVYS7TYUILMTF5R7TQXXH5RRHDLFQMQAKM66WLA, seg_pos={0 0}, upload=256935]: Uploading piece... Nov 21 11:49:17.244784930 (291h45m15.796255391s) [noise=true , address=145.79.227.10:4403 , piece_id=XS5LV3OLSHZWHVBGVPFV4TFGBPZGHMLGNF6OGVWWQQMTXVWC5QCQ, seg_pos={0 0}, upload=256935]: Uploading piece... Nov 21 11:49:17.245522555 (291h45m15.796993016s) [noise=true , address=77.90.31.13:4628 , piece_id=T3KCZD5JV6FAEG6VHBK6HG2E2C6XMP4A6NVAWFWKZTWJIA57L4AA, seg_pos={0 0}, upload=256935]: Uploading piece... Nov 21 11:49:17.245623898 (291h45m15.797094359s) [noise=true , address=145.79.227.8:4330 , piece_id=ZCJFYOK2Q2B6DDNNWGW6TXFPUJMPOGLV6HAPQO2HEVCGXAQVGBRA, seg_pos={0 0}, upload=256935]: Uploading piece... Nov 21 11:49:17.246035040 (291h45m15.797505491s) [noise=true , address=77.90.31.15:4698 , piece_id=QOPKTWSMOAHGWQ3XVTFJIBMWTJTZ3PJLI6U7BF4TDWR3ZTM2RG5Q, seg_pos={0 0}, upload=256935]: Uploading piece... Nov 21 11:49:17.246184175 (291h45m15.797654636s) [noise=true , address=45.91.116.4:4937 , piece_id=OKQ3DD3KZ73K3RRAE4RR6EG6HOYWZYGGJPKN2QTVN3BWWNKK7PDQ, seg_pos={0 0}, upload=256935]: Uploading piece... Nov 21 11:49:17.246797051 (291h45m15.798267522s) [noise=true , address=45.91.116.10:4081 , piece_id=CSMEKQE6SUSZ52HLGWVDXINNPRIYR6LJM5FBOI5RNFVH7VF7OLBQ, seg_pos={0 0}, upload=256935]: Uploading piece... Nov 21 11:49:17.247334054 (291h45m15.798804515s) [noise=true , address=77.90.31.14:4663 , piece_id=CECM54T2XTGYW4BFN4TZY2DYUH47J6YO6PGF2QQ6SOCUMLSW6XMQ, seg_pos={0 0}, upload=256935]: Uploading piece... Nov 21 11:49:17.247548422 (291h45m15.799018873s) [noise=true , address=145.79.227.30:4761 , piece_id=SDNZAHFY6GR7I4L6BTFGXSAKA5O5MHJ54NHMA77VTAXTCWQQSFQQ, seg_pos={0 0}, upload=256935]: Uploading piece... Nov 21 11:49:17.249260893 (291h45m15.800731354s) [noise=true , address=145.79.227.23:4546 , piece_id=HIT5PPIDXBOIXNY6X3UPCJULOM7YX7BI4VST4IEP6VO7NPK7YMDQ, seg_pos={0 0}, upload=256935]: Uploading piece... Nov 21 11:49:17.249559843 (291h45m15.801030304s) [noise=true , address=145.79.227.28:4726 , piece_id=7XZ277AZOYL6KFSIAAHNTWNZ2PBWMT2YIQFRWNTYEIIINQ6VTRUQ, seg_pos={0 0}, upload=256935]: Uploading piece... Nov 21 11:49:17.251617881 (291h45m15.803088333s) [noise=true , address=145.79.227.31:4798 , piece_id=VKGEFEBHEEU5VUVIVEV7OVY3SUPUXFVGNP7OWNJSBOQAKDYQFQGQ, seg_pos={0 0}, upload=256935]: Uploading piece... Nov 21 11:49:17.252868232 (291h45m15.804338704s) [noise=true , address=145.79.227.8:4331 , piece_id=GK4K577UUQ7IM2NNN5Q4BCJ26JK3EET6VNEEWOPQO7I5VCLOGNFQ, seg_pos={0 0}, upload=256935]: Uploading piece... Nov 21 11:49:17.254615760 (291h45m15.806086221s) [noise=true , address=45.91.116.8:4048 , piece_id=N7NO3DOXYOQDEAGQ43OPWR3E5CBYZX5J4A4EJV2VLGKOKJBXCIQQ, seg_pos={0 0}, upload=256935]: Uploading piece... Nov 21 11:49:18.965747830 (291h45m17.517218281s) [upload=256935]: Next returned nil segment Nov 21 11:49:18.965769751 (291h45m17.517240243s) [upload=256935]: Waiting for error group managing segments... Nov 21 11:49:19.250399321 (291h45m17.801869792s) [noise=true , address=145.79.227.30:4761 , piece_id=SDNZAHFY6GR7I4L6BTFGXSAKA5O5MHJ54NHMA77VTAXTCWQQSFQQ, seg_pos={0 0}, upload=256935]: Done uploading piece. err: <nil> Nov 21 11:49:19.250854327 (291h45m17.802324788s) [noise=true , address=145.79.227.10:4403 , piece_id=XS5LV3OLSHZWHVBGVPFV4TFGBPZGHMLGNF6OGVWWQQMTXVWC5QCQ, seg_pos={0 0}, upload=256935]: Done uploading piece. err: <nil> Nov 21 11:49:19.260959880 (291h45m17.812430341s) [noise=true , address=145.79.227.11:4731 , piece_id=2NO5DRKFPM22XIWMQU4GDYHKXWYQX2QRAYP3YQBSIWTLAMW6LAPA, seg_pos={0 0}, upload=256935]: Done uploading piece. err: <nil> Nov 21 11:49:19.262082247 (291h45m17.813552709s) [noise=true , address=145.79.227.8:4330 , piece_id=ZCJFYOK2Q2B6DDNNWGW6TXFPUJMPOGLV6HAPQO2HEVCGXAQVGBRA, seg_pos={0 0}, upload=256935]: Done uploading piece. err: <nil> Nov 21 11:49:19.268139792 (291h45m17.819610263s) [noise=true , address=145.79.227.28:4726 , piece_id=7XZ277AZOYL6KFSIAAHNTWNZ2PBWMT2YIQFRWNTYEIIINQ6VTRUQ, seg_pos={0 0}, upload=256935]: Done uploading piece. err: <nil> Nov 21 11:49:19.280973680 (291h45m17.832444141s) [noise=true , address=145.79.227.31:4798 , piece_id=VKGEFEBHEEU5VUVIVEV7OVY3SUPUXFVGNP7OWNJSBOQAKDYQFQGQ, seg_pos={0 0}, upload=256935]: Done uploading piece. err: <nil> Nov 21 11:49:19.282868778 (291h45m17.83433924s) [noise=true , address=145.79.227.23:4546 , piece_id=HIT5PPIDXBOIXNY6X3UPCJULOM7YX7BI4VST4IEP6VO7NPK7YMDQ, seg_pos={0 0}, upload=256935]: Done uploading piece. err: <nil> Nov 21 11:49:19.286078351 (291h45m17.837548812s) [noise=true , address=145.79.227.8:4331 , piece_id=GK4K577UUQ7IM2NNN5Q4BCJ26JK3EET6VNEEWOPQO7I5VCLOGNFQ, seg_pos={0 0}, upload=256935]: Done uploading piece. err: <nil> Nov 21 11:49:19.821632116 (291h45m18.373102577s) [noise=true , address=77.90.31.13:4628 , piece_id=T3KCZD5JV6FAEG6VHBK6HG2E2C6XMP4A6NVAWFWKZTWJIA57L4AA, seg_pos={0 0}, upload=256935]: Done uploading piece. err: <nil> Nov 21 11:49:19.824423853 (291h45m18.375894304s) [noise=true , address=77.90.31.14:4663 , piece_id=CECM54T2XTGYW4BFN4TZY2DYUH47J6YO6PGF2QQ6SOCUMLSW6XMQ, seg_pos={0 0}, upload=256935]: Done uploading piece. err: <nil> Nov 21 11:49:19.850707311 (291h45m18.402177772s) [noise=true , address=45.91.116.8:4048 , piece_id=N7NO3DOXYOQDEAGQ43OPWR3E5CBYZX5J4A4EJV2VLGKOKJBXCIQQ, seg_pos={0 0}, upload=256935]: Done uploading piece. err: <nil> Nov 21 11:49:19.908423644 (291h45m18.459894105s) [noise=true , address=77.90.31.11:4555 , piece_id=HLV5EPKQNPRWQKHNJLB5ATKRWSS6XW3HMWYXUJL4L7VM7VL4MBZQ, seg_pos={0 0}, upload=256935]: Done uploading piece. err: <nil> Nov 21 11:49:19.958255192 (291h45m18.509725653s) [noise=true , address=45.91.116.14:4191 , piece_id=3TMBBNLJQ4OZ7RVYS7TYUILMTF5R7TQXXH5RRHDLFQMQAKM66WLA, seg_pos={0 0}, upload=256935]: Done uploading piece. err: <nil> Nov 21 11:49:19.958317902 (291h45m18.509788353s) [seg_pos={0 0}, upload=256935]: Segment reached optimal threshold of 13 pieces. Nov 21 11:49:19.958597444 (291h45m18.510067905s) [noise=true , address=77.90.31.15:4698 , piece_id=QOPKTWSMOAHGWQ3XVTFJIBMWTJTZ3PJLI6U7BF4TDWR3ZTM2RG5Q, seg_pos={0 0}, upload=256935]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj): protocol: write tcp 188.166.225.170:49482->77.90.31.15:4698: use of closed network connection; write tcp 188.166.225.170:49482->77.90.31.15:4698: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49482->77.90.31.15:4698: use of closed network connection Nov 21 11:49:19.958607714 (291h45m18.510078175s) [noise=true , address=77.90.31.15:4698 , piece_id=QOPKTWSMOAHGWQ3XVTFJIBMWTJTZ3PJLI6U7BF4TDWR3ZTM2RG5Q, seg_pos={0 0}, upload=256935]: Long tail cancellation due to optimal threshold Nov 21 11:49:19.958655214 (291h45m18.510125675s) [noise=true , address=45.91.116.10:4081 , piece_id=CSMEKQE6SUSZ52HLGWVDXINNPRIYR6LJM5FBOI5RNFVH7VF7OLBQ, seg_pos={0 0}, upload=256935]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG): protocol: write tcp 188.166.225.170:57522->45.91.116.10:4081: use of closed network connection; write tcp 188.166.225.170:57522->45.91.116.10:4081: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57522->45.91.116.10:4081: use of closed network connection Nov 21 11:49:19.958672887 (291h45m18.510143349s) [noise=true , address=45.91.116.10:4081 , piece_id=CSMEKQE6SUSZ52HLGWVDXINNPRIYR6LJM5FBOI5RNFVH7VF7OLBQ, seg_pos={0 0}, upload=256935]: Long tail cancellation due to optimal threshold Nov 21 11:49:19.958915429 (291h45m18.5103859s) [noise=true , address=45.91.116.4:4937 , piece_id=OKQ3DD3KZ73K3RRAE4RR6EG6HOYWZYGGJPKN2QTVN3BWWNKK7PDQ, seg_pos={0 0}, upload=256935]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq): protocol: expected piece hash; context canceled; EOF Nov 21 11:49:19.958928194 (291h45m18.510398655s) [noise=true , address=45.91.116.4:4937 , piece_id=OKQ3DD3KZ73K3RRAE4RR6EG6HOYWZYGGJPKN2QTVN3BWWNKK7PDQ, seg_pos={0 0}, upload=256935]: Long tail cancellation due to optimal threshold Nov 21 11:49:19.958967489 (291h45m18.51043795s) [seg_pos={0 0}, upload=256935]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 21 11:49:19.959064374 (291h45m18.510534835s) [upload=256935]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 21 11:49:19.959071227 (291h45m18.510541688s) [upload=256935]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 21 11:49:20.260011595 (291h45m18.811482076s) [upload=256935]: Done uploading segments... Nov 21 11:49:20.260040470 (291h45m18.811510931s) [upload=256935]: Upload finished. err: <nil> Duration: 6.343441355s core uplink us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 2 out of 60 is failed (threshold=3) [...................X.X......................................] duration: 5.287695967s trace: https://jaeger.dev.storj.io/trace/a7f69ee106efb7f FAIL (2025-11-21 11:51:33 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== upload=256586]: Uploading piece... Nov 21 11:25:27.210346024 (291h21m25.761816485s) [noise=true , address=89.149.201.147:4008 , piece_id=HCX7I4ICPEHDWTYDFIQMKZ5EE5CGPQLYAIHF7WZMTUUU6CNZPHTA, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.211425289 (291h21m25.76289575s) [noise=true , address=81.171.14.70:4044 , piece_id=QYUBNGS2FAFHIWMW7GAGYJ3UVIWVP62LIDCOK6QUXEIQNIXRJDAQ, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.213103795 (291h21m25.764574246s) [noise=true , address=185.167.204.205:4022 , piece_id=BES3YK6PYQV2AKOWSAGHF7DGGWGWSWPIJU2ICAQ5WBHKZ5VT5L6A, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.213408395 (291h21m25.764878856s) [noise=true , address=185.167.204.205:4027 , piece_id=IVPONKBF2PIHZQAW3L4YA4HOMQV5BQML6SKKCJLIEGT5SZSQ2AOA, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.214501747 (291h21m25.765972208s) [noise=true , address=81.171.14.70:4042 , piece_id=XNREZEUZ5L5KZ3L7ZOL6IRK7IHU6DJN4RLEQZRFMDKDH2JWC24CQ, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.215013752 (291h21m25.766484213s) [noise=true , address=81.171.14.70:4034 , piece_id=ITRLKYVNFSYBLABNGNFO3HRJNMX3MWB345FIMWSIPBNAVZPA47UA, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.215087572 (291h21m25.766558044s) [noise=true , address=185.167.204.205:4031 , piece_id=ZEI2YSIQC655R7WS43WADXGFT5RKT5LFSN6IZEQ6SHTDVOV7BZBQ, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.216432844 (291h21m25.767903305s) [noise=true , address=185.167.204.206:4056 , piece_id=EAMZIK6AVJL7IMFF5VOLW3SMCGYKSKB2VXJTNUNLJCVIT7EKXNMQ, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.216662281 (291h21m25.768132742s) [noise=true , address=89.149.201.147:4011 , piece_id=ZATNMD5HOK35AWGJJ5V66VDJ4YGUVZQJ2OBERQL32ZV5GW2HBSMQ, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.218008314 (291h21m25.769478775s) [noise=true , address=81.171.14.70:4032 , piece_id=CFH2RTR3BLBUB5ZIZCFC6UHT3FHT3B2XSCHQMKXNGDJJGPHEMCTQ, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:28.035150976 (291h21m26.586621437s) [upload=256586]: Next returned nil segment Nov 21 11:25:28.035169622 (291h21m26.586640083s) [upload=256586]: Waiting for error group managing segments... Nov 21 11:25:28.416646836 (291h21m26.968117317s) [noise=true , address=185.167.204.206:4064 , piece_id=SRFIWFQLRAGCNIKYLHMGT4QRUL3WH2QPLLIKAJ7DLFJEQUW5DPPA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.417338854 (291h21m26.968809325s) [noise=true , address=185.167.204.205:4033 , piece_id=FEWQVHPIKN3Q6O5AZ6AGJBLUPZ34PBZFB5X3XNSKGPOGI4AIO7JA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.421089406 (291h21m26.972559867s) [noise=true , address=185.167.204.206:4072 , piece_id=GYCO53OWGOMOHK4FHC5GENN4LTQGSZKZKOI5V534RLV3LHLPKC6Q, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.426252887 (291h21m26.977723348s) [noise=true , address=185.167.204.206:4046 , piece_id=BZ57ZA35ITEOA6CAZJGNODK5MUTXEZWDV7E2OGBZOSMS4OQZYAEA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.431720689 (291h21m26.98319115s) [noise=true , address=185.167.204.205:4027 , piece_id=IVPONKBF2PIHZQAW3L4YA4HOMQV5BQML6SKKCJLIEGT5SZSQ2AOA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.438243951 (291h21m26.989714412s) [noise=true , address=185.167.204.205:4022 , piece_id=BES3YK6PYQV2AKOWSAGHF7DGGWGWSWPIJU2ICAQ5WBHKZ5VT5L6A, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.443364331 (291h21m26.994834792s) [noise=true , address=185.167.204.206:4056 , piece_id=EAMZIK6AVJL7IMFF5VOLW3SMCGYKSKB2VXJTNUNLJCVIT7EKXNMQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.445594728 (291h21m26.997065179s) [noise=true , address=185.167.204.205:4016 , piece_id=EHQS3546UXRXTL2MGFCSRTPDDPAYXEYLMISPP5AWY337MHX7HFLQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.449325822 (291h21m27.000796283s) [noise=true , address=185.167.204.205:4031 , piece_id=ZEI2YSIQC655R7WS43WADXGFT5RKT5LFSN6IZEQ6SHTDVOV7BZBQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.449761211 (291h21m27.001231672s) [noise=true , address=185.167.204.205:4018 , piece_id=7OSLPWJLKFTIJPZEAFMZOAH6JAAMX6HKROI25FODFYBPNJWSMLWA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.450419845 (291h21m27.001890306s) [noise=true , address=185.167.204.206:4062 , piece_id=UF3S6NAVGS2KQJPGA5WXP7YG4RQO44P445SNPCWXKXMONZAYUTNQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.453201252 (291h21m27.004671713s) [noise=true , address=185.167.204.205:4007 , piece_id=MEBGOPZ6MFMQY4UZARS2LKJKEMNN5YGSLKPJGTWSKS45CUNXSRGQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:29.661209626 (291h21m28.212680087s) [noise=true , address=89.149.201.147:4016 , piece_id=TGRWGGGBKDANC7YKROWPRMIYNAVFEVE3RXOSUOINFI7KTRS2IYUQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:29.693169182 (291h21m28.244639643s) [noise=true , address=89.149.201.147:4002 , piece_id=UIYJSCMY3LCLRHZUJB6JFJHHXKMYTVKERTZW7F7X44MWJETPY6XA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:29.721204497 (291h21m28.272674958s) [noise=true , address=89.149.201.147:4008 , piece_id=HCX7I4ICPEHDWTYDFIQMKZ5EE5CGPQLYAIHF7WZMTUUU6CNZPHTA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:29.764584711 (291h21m28.316055172s) [noise=true , address=89.149.201.147:4001 , piece_id=QUS6NB3NQBLJFB6YCK3EBDPKCTWEWGKAT3NOR24TVI65SQKHWDOQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:29.768284075 (291h21m28.319754536s) [noise=true , address=89.149.201.147:4011 , piece_id=ZATNMD5HOK35AWGJJ5V66VDJ4YGUVZQJ2OBERQL32ZV5GW2HBSMQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:29.789681961 (291h21m28.341152422s) [noise=true , address=89.149.201.147:4003 , piece_id=RUAYGN2RGF5JJTWFEGIHWH3BHEUDSS5DN2QJ6YLSHPF5BGUXD3RA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:37.298598682 (291h21m35.850069143s) [noise=true , address=81.171.14.70:4043 , piece_id=T3NF6JR3VJIGDMPNUBLMJ3AOTQP5IHOGJYORMCSS7ZMOENQETZ7A, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:37.497478356 (291h21m36.048948818s) [noise=true , address=81.171.14.70:4032 , piece_id=CFH2RTR3BLBUB5ZIZCFC6UHT3FHT3B2XSCHQMKXNGDJJGPHEMCTQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:40.406166933 (291h21m38.957637394s) [noise=true , address=81.171.14.70:4045 , piece_id=BRWQ7G3E547PJE2RLGSESIB3HMP5C4M5NSKAJNGWSFXUBEO6SYEQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:42.511041157 (291h21m41.062511608s) [noise=true , address=81.171.14.70:4029 , piece_id=DKJM5E7EY7LOSBSEBQOZRC6CI5ZPPE47HFOFQJROPCUMUV567CAA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:26:12.012682521 (291h22m10.564152982s) [noise=true , address=81.171.14.70:4042 , piece_id=XNREZEUZ5L5KZ3L7ZOL6IRK7IHU6DJN4RLEQZRFMDKDH2JWC24CQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:26:15.596046283 (291h22m14.147516744s) [noise=true , address=81.171.14.70:4033 , piece_id=JRKPD4EP6FZM5HISOIDJYZY7FAWWQCADAS47FGW2JM5A7RFJR2BQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:26:18.453059659 (291h22m17.00453012s) [noise=true , address=81.171.14.70:4046 , piece_id=OSONRSF6KJS7F2QJVCC3FPZK4AV7QPG5YSM47CW5LIW7EME44AWA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:26:23.442583344 (291h22m21.994053805s) [noise=true , address=81.171.14.70:4034 , piece_id=ITRLKYVNFSYBLABNGNFO3HRJNMX3MWB345FIMWSIPBNAVZPA47UA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:26:23.442621967 (291h22m21.994092428s) [seg_pos={0 0}, upload=256586]: Segment reached optimal threshold of 26 pieces. Nov 21 11:26:23.442836976 (291h22m21.994307437s) [noise=true , address=81.171.14.70:4044 , piece_id=QYUBNGS2FAFHIWMW7GAGYJ3UVIWVP62LIDCOK6QUXEIQNIXRJDAQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1MUVGecysnTSFcJoHUL7QMWb3h6dtXgUd7FbA672fQCyEWhePr): protocol: expected piece hash; context canceled; EOF Nov 21 11:26:23.442845934 (291h22m21.994316395s) [noise=true , address=81.171.14.70:4044 , piece_id=QYUBNGS2FAFHIWMW7GAGYJ3UVIWVP62LIDCOK6QUXEIQNIXRJDAQ, seg_pos={0 0}, upload=256586]: Long tail cancellation due to optimal threshold Nov 21 11:26:23.443227491 (291h22m21.994697942s) [noise=true , address=81.171.14.70:4027 , piece_id=4FDFCCWF3EIROQAFLOHGS743523OYGSXWQSPQEVD3AMQ6WKDQI7Q, seg_pos={0 0}, upload=256586]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122cYvgTxSvC2aEXnBkRk2o8ARTBXj9gqPoa12vW8hogSN6kpA4): protocol: expected piece hash; context canceled; EOF Nov 21 11:26:23.443237770 (291h22m21.994708231s) [noise=true , address=81.171.14.70:4027 , piece_id=4FDFCCWF3EIROQAFLOHGS743523OYGSXWQSPQEVD3AMQ6WKDQI7Q, seg_pos={0 0}, upload=256586]: Long tail cancellation due to optimal threshold Nov 21 11:26:23.443274720 (291h22m21.994745181s) [seg_pos={0 0}, upload=256586]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Nov 21 11:26:23.443378929 (291h22m21.99484939s) [upload=256586]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 21 11:26:23.443387044 (291h22m21.994857495s) [upload=256586]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 21 11:26:23.685139626 (291h22m22.236610077s) [upload=256586]: Done uploading segments... Nov 21 11:26:23.685160676 (291h22m22.236631127s) [upload=256586]: Upload finished. err: <nil> Duration: 5.287723541s core uplink eu-select-1 eu1-placement-15
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 42 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX..................] duration: 3.603489297s trace: https://jaeger.dev.storj.io/trace/6af15a3a67b655d5 FAIL (2025-11-21 11:51:22 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 3.603515297s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 16 out of 60 is failed (threshold=3) [..X.X......X.XX...X.X.X....X............X....X.X.X..X.X..X..] duration: 6.330563932s trace: https://jaeger.dev.storj.io/trace/34d009b2527f9379 FAIL (2025-11-21 11:50:51 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB, Piece ID: N7NO3DOXYOQDEAGQ43OPWR3E5CBYZX5J4A4EJV2VLGKOKJBXCIQQ) context deadline exceeded; piecestore: (Node ID: 1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ, Piece ID: HLV5EPKQNPRWQKHNJLB5ATKRWSS6XW3HMWYXUJL4L7VM7VL4MBZQ) context deadline exceeded; piecestore: (Node ID: 1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, Piece ID: T3KCZD5JV6FAEG6VHBK6HG2E2C6XMP4A6NVAWFWKZTWJIA57L4AA) context deadline exceeded; piecestore: (Node ID: 155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, Piece ID: CECM54T2XTGYW4BFN4TZY2DYUH47J6YO6PGF2QQ6SOCUMLSW6XMQ) context deadline exceeded ======LOGS====== ments... Nov 21 11:49:16.994612805 (291h45m15.546083266s) [upload=256935]: Got next segment. Inline: false Nov 21 11:49:16.994647011 (291h45m15.546117472s) [upload=256935]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 21 11:49:16.994651759 (291h45m15.54612221s) [upload=256935]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 21 11:49:17.243148153 (291h45m15.794618604s) [seg_pos={0 0}, upload=256935]: Begin upload segment... Nov 21 11:49:17.243197467 (291h45m15.794667929s) [seg_pos={0 0}, upload=256935]: Done begin upload segment. Nov 21 11:49:17.243223908 (291h45m15.794694369s) [noise=true , address=77.90.31.11:4555 , piece_id=HLV5EPKQNPRWQKHNJLB5ATKRWSS6XW3HMWYXUJL4L7VM7VL4MBZQ, seg_pos={0 0}, upload=256935]: Uploading piece... Nov 21 11:49:17.243288660 (291h45m15.794759112s) [noise=true , address=145.79.227.11:4731 , piece_id=2NO5DRKFPM22XIWMQU4GDYHKXWYQX2QRAYP3YQBSIWTLAMW6LAPA, seg_pos={0 0}, upload=256935]: Uploading piece... Nov 21 11:49:17.243999364 (291h45m15.795469815s) [noise=true , address=45.91.116.14:4191 , piece_id=3TMBBNLJQ4OZ7RVYS7TYUILMTF5R7TQXXH5RRHDLFQMQAKM66WLA, seg_pos={0 0}, upload=256935]: Uploading piece... Nov 21 11:49:17.244784930 (291h45m15.796255391s) [noise=true , address=145.79.227.10:4403 , piece_id=XS5LV3OLSHZWHVBGVPFV4TFGBPZGHMLGNF6OGVWWQQMTXVWC5QCQ, seg_pos={0 0}, upload=256935]: Uploading piece... Nov 21 11:49:17.245522555 (291h45m15.796993016s) [noise=true , address=77.90.31.13:4628 , piece_id=T3KCZD5JV6FAEG6VHBK6HG2E2C6XMP4A6NVAWFWKZTWJIA57L4AA, seg_pos={0 0}, upload=256935]: Uploading piece... Nov 21 11:49:17.245623898 (291h45m15.797094359s) [noise=true , address=145.79.227.8:4330 , piece_id=ZCJFYOK2Q2B6DDNNWGW6TXFPUJMPOGLV6HAPQO2HEVCGXAQVGBRA, seg_pos={0 0}, upload=256935]: Uploading piece... Nov 21 11:49:17.246035040 (291h45m15.797505491s) [noise=true , address=77.90.31.15:4698 , piece_id=QOPKTWSMOAHGWQ3XVTFJIBMWTJTZ3PJLI6U7BF4TDWR3ZTM2RG5Q, seg_pos={0 0}, upload=256935]: Uploading piece... Nov 21 11:49:17.246184175 (291h45m15.797654636s) [noise=true , address=45.91.116.4:4937 , piece_id=OKQ3DD3KZ73K3RRAE4RR6EG6HOYWZYGGJPKN2QTVN3BWWNKK7PDQ, seg_pos={0 0}, upload=256935]: Uploading piece... Nov 21 11:49:17.246797051 (291h45m15.798267522s) [noise=true , address=45.91.116.10:4081 , piece_id=CSMEKQE6SUSZ52HLGWVDXINNPRIYR6LJM5FBOI5RNFVH7VF7OLBQ, seg_pos={0 0}, upload=256935]: Uploading piece... Nov 21 11:49:17.247334054 (291h45m15.798804515s) [noise=true , address=77.90.31.14:4663 , piece_id=CECM54T2XTGYW4BFN4TZY2DYUH47J6YO6PGF2QQ6SOCUMLSW6XMQ, seg_pos={0 0}, upload=256935]: Uploading piece... Nov 21 11:49:17.247548422 (291h45m15.799018873s) [noise=true , address=145.79.227.30:4761 , piece_id=SDNZAHFY6GR7I4L6BTFGXSAKA5O5MHJ54NHMA77VTAXTCWQQSFQQ, seg_pos={0 0}, upload=256935]: Uploading piece... Nov 21 11:49:17.249260893 (291h45m15.800731354s) [noise=true , address=145.79.227.23:4546 , piece_id=HIT5PPIDXBOIXNY6X3UPCJULOM7YX7BI4VST4IEP6VO7NPK7YMDQ, seg_pos={0 0}, upload=256935]: Uploading piece... Nov 21 11:49:17.249559843 (291h45m15.801030304s) [noise=true , address=145.79.227.28:4726 , piece_id=7XZ277AZOYL6KFSIAAHNTWNZ2PBWMT2YIQFRWNTYEIIINQ6VTRUQ, seg_pos={0 0}, upload=256935]: Uploading piece... Nov 21 11:49:17.251617881 (291h45m15.803088333s) [noise=true , address=145.79.227.31:4798 , piece_id=VKGEFEBHEEU5VUVIVEV7OVY3SUPUXFVGNP7OWNJSBOQAKDYQFQGQ, seg_pos={0 0}, upload=256935]: Uploading piece... Nov 21 11:49:17.252868232 (291h45m15.804338704s) [noise=true , address=145.79.227.8:4331 , piece_id=GK4K577UUQ7IM2NNN5Q4BCJ26JK3EET6VNEEWOPQO7I5VCLOGNFQ, seg_pos={0 0}, upload=256935]: Uploading piece... Nov 21 11:49:17.254615760 (291h45m15.806086221s) [noise=true , address=45.91.116.8:4048 , piece_id=N7NO3DOXYOQDEAGQ43OPWR3E5CBYZX5J4A4EJV2VLGKOKJBXCIQQ, seg_pos={0 0}, upload=256935]: Uploading piece... Nov 21 11:49:18.965747830 (291h45m17.517218281s) [upload=256935]: Next returned nil segment Nov 21 11:49:18.965769751 (291h45m17.517240243s) [upload=256935]: Waiting for error group managing segments... Nov 21 11:49:19.250399321 (291h45m17.801869792s) [noise=true , address=145.79.227.30:4761 , piece_id=SDNZAHFY6GR7I4L6BTFGXSAKA5O5MHJ54NHMA77VTAXTCWQQSFQQ, seg_pos={0 0}, upload=256935]: Done uploading piece. err: <nil> Nov 21 11:49:19.250854327 (291h45m17.802324788s) [noise=true , address=145.79.227.10:4403 , piece_id=XS5LV3OLSHZWHVBGVPFV4TFGBPZGHMLGNF6OGVWWQQMTXVWC5QCQ, seg_pos={0 0}, upload=256935]: Done uploading piece. err: <nil> Nov 21 11:49:19.260959880 (291h45m17.812430341s) [noise=true , address=145.79.227.11:4731 , piece_id=2NO5DRKFPM22XIWMQU4GDYHKXWYQX2QRAYP3YQBSIWTLAMW6LAPA, seg_pos={0 0}, upload=256935]: Done uploading piece. err: <nil> Nov 21 11:49:19.262082247 (291h45m17.813552709s) [noise=true , address=145.79.227.8:4330 , piece_id=ZCJFYOK2Q2B6DDNNWGW6TXFPUJMPOGLV6HAPQO2HEVCGXAQVGBRA, seg_pos={0 0}, upload=256935]: Done uploading piece. err: <nil> Nov 21 11:49:19.268139792 (291h45m17.819610263s) [noise=true , address=145.79.227.28:4726 , piece_id=7XZ277AZOYL6KFSIAAHNTWNZ2PBWMT2YIQFRWNTYEIIINQ6VTRUQ, seg_pos={0 0}, upload=256935]: Done uploading piece. err: <nil> Nov 21 11:49:19.280973680 (291h45m17.832444141s) [noise=true , address=145.79.227.31:4798 , piece_id=VKGEFEBHEEU5VUVIVEV7OVY3SUPUXFVGNP7OWNJSBOQAKDYQFQGQ, seg_pos={0 0}, upload=256935]: Done uploading piece. err: <nil> Nov 21 11:49:19.282868778 (291h45m17.83433924s) [noise=true , address=145.79.227.23:4546 , piece_id=HIT5PPIDXBOIXNY6X3UPCJULOM7YX7BI4VST4IEP6VO7NPK7YMDQ, seg_pos={0 0}, upload=256935]: Done uploading piece. err: <nil> Nov 21 11:49:19.286078351 (291h45m17.837548812s) [noise=true , address=145.79.227.8:4331 , piece_id=GK4K577UUQ7IM2NNN5Q4BCJ26JK3EET6VNEEWOPQO7I5VCLOGNFQ, seg_pos={0 0}, upload=256935]: Done uploading piece. err: <nil> Nov 21 11:49:19.821632116 (291h45m18.373102577s) [noise=true , address=77.90.31.13:4628 , piece_id=T3KCZD5JV6FAEG6VHBK6HG2E2C6XMP4A6NVAWFWKZTWJIA57L4AA, seg_pos={0 0}, upload=256935]: Done uploading piece. err: <nil> Nov 21 11:49:19.824423853 (291h45m18.375894304s) [noise=true , address=77.90.31.14:4663 , piece_id=CECM54T2XTGYW4BFN4TZY2DYUH47J6YO6PGF2QQ6SOCUMLSW6XMQ, seg_pos={0 0}, upload=256935]: Done uploading piece. err: <nil> Nov 21 11:49:19.850707311 (291h45m18.402177772s) [noise=true , address=45.91.116.8:4048 , piece_id=N7NO3DOXYOQDEAGQ43OPWR3E5CBYZX5J4A4EJV2VLGKOKJBXCIQQ, seg_pos={0 0}, upload=256935]: Done uploading piece. err: <nil> Nov 21 11:49:19.908423644 (291h45m18.459894105s) [noise=true , address=77.90.31.11:4555 , piece_id=HLV5EPKQNPRWQKHNJLB5ATKRWSS6XW3HMWYXUJL4L7VM7VL4MBZQ, seg_pos={0 0}, upload=256935]: Done uploading piece. err: <nil> Nov 21 11:49:19.958255192 (291h45m18.509725653s) [noise=true , address=45.91.116.14:4191 , piece_id=3TMBBNLJQ4OZ7RVYS7TYUILMTF5R7TQXXH5RRHDLFQMQAKM66WLA, seg_pos={0 0}, upload=256935]: Done uploading piece. err: <nil> Nov 21 11:49:19.958317902 (291h45m18.509788353s) [seg_pos={0 0}, upload=256935]: Segment reached optimal threshold of 13 pieces. Nov 21 11:49:19.958597444 (291h45m18.510067905s) [noise=true , address=77.90.31.15:4698 , piece_id=QOPKTWSMOAHGWQ3XVTFJIBMWTJTZ3PJLI6U7BF4TDWR3ZTM2RG5Q, seg_pos={0 0}, upload=256935]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj): protocol: write tcp 188.166.225.170:49482->77.90.31.15:4698: use of closed network connection; write tcp 188.166.225.170:49482->77.90.31.15:4698: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49482->77.90.31.15:4698: use of closed network connection Nov 21 11:49:19.958607714 (291h45m18.510078175s) [noise=true , address=77.90.31.15:4698 , piece_id=QOPKTWSMOAHGWQ3XVTFJIBMWTJTZ3PJLI6U7BF4TDWR3ZTM2RG5Q, seg_pos={0 0}, upload=256935]: Long tail cancellation due to optimal threshold Nov 21 11:49:19.958655214 (291h45m18.510125675s) [noise=true , address=45.91.116.10:4081 , piece_id=CSMEKQE6SUSZ52HLGWVDXINNPRIYR6LJM5FBOI5RNFVH7VF7OLBQ, seg_pos={0 0}, upload=256935]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG): protocol: write tcp 188.166.225.170:57522->45.91.116.10:4081: use of closed network connection; write tcp 188.166.225.170:57522->45.91.116.10:4081: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57522->45.91.116.10:4081: use of closed network connection Nov 21 11:49:19.958672887 (291h45m18.510143349s) [noise=true , address=45.91.116.10:4081 , piece_id=CSMEKQE6SUSZ52HLGWVDXINNPRIYR6LJM5FBOI5RNFVH7VF7OLBQ, seg_pos={0 0}, upload=256935]: Long tail cancellation due to optimal threshold Nov 21 11:49:19.958915429 (291h45m18.5103859s) [noise=true , address=45.91.116.4:4937 , piece_id=OKQ3DD3KZ73K3RRAE4RR6EG6HOYWZYGGJPKN2QTVN3BWWNKK7PDQ, seg_pos={0 0}, upload=256935]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq): protocol: expected piece hash; context canceled; EOF Nov 21 11:49:19.958928194 (291h45m18.510398655s) [noise=true , address=45.91.116.4:4937 , piece_id=OKQ3DD3KZ73K3RRAE4RR6EG6HOYWZYGGJPKN2QTVN3BWWNKK7PDQ, seg_pos={0 0}, upload=256935]: Long tail cancellation due to optimal threshold Nov 21 11:49:19.958967489 (291h45m18.51043795s) [seg_pos={0 0}, upload=256935]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 21 11:49:19.959064374 (291h45m18.510534835s) [upload=256935]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 21 11:49:19.959071227 (291h45m18.510541688s) [upload=256935]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 21 11:49:20.260011595 (291h45m18.811482076s) [upload=256935]: Done uploading segments... Nov 21 11:49:20.260040470 (291h45m18.811510931s) [upload=256935]: Upload finished. err: <nil> Duration: 6.33059497s core uplink us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 2 out of 60 is failed (threshold=3) [....................X.X.....................................] duration: 7.271589155s trace: https://jaeger.dev.storj.io/trace/a7f69ee106efb7f FAIL (2025-11-21 11:50:37 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== upload=256586]: Uploading piece... Nov 21 11:25:27.210346024 (291h21m25.761816485s) [noise=true , address=89.149.201.147:4008 , piece_id=HCX7I4ICPEHDWTYDFIQMKZ5EE5CGPQLYAIHF7WZMTUUU6CNZPHTA, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.211425289 (291h21m25.76289575s) [noise=true , address=81.171.14.70:4044 , piece_id=QYUBNGS2FAFHIWMW7GAGYJ3UVIWVP62LIDCOK6QUXEIQNIXRJDAQ, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.213103795 (291h21m25.764574246s) [noise=true , address=185.167.204.205:4022 , piece_id=BES3YK6PYQV2AKOWSAGHF7DGGWGWSWPIJU2ICAQ5WBHKZ5VT5L6A, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.213408395 (291h21m25.764878856s) [noise=true , address=185.167.204.205:4027 , piece_id=IVPONKBF2PIHZQAW3L4YA4HOMQV5BQML6SKKCJLIEGT5SZSQ2AOA, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.214501747 (291h21m25.765972208s) [noise=true , address=81.171.14.70:4042 , piece_id=XNREZEUZ5L5KZ3L7ZOL6IRK7IHU6DJN4RLEQZRFMDKDH2JWC24CQ, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.215013752 (291h21m25.766484213s) [noise=true , address=81.171.14.70:4034 , piece_id=ITRLKYVNFSYBLABNGNFO3HRJNMX3MWB345FIMWSIPBNAVZPA47UA, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.215087572 (291h21m25.766558044s) [noise=true , address=185.167.204.205:4031 , piece_id=ZEI2YSIQC655R7WS43WADXGFT5RKT5LFSN6IZEQ6SHTDVOV7BZBQ, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.216432844 (291h21m25.767903305s) [noise=true , address=185.167.204.206:4056 , piece_id=EAMZIK6AVJL7IMFF5VOLW3SMCGYKSKB2VXJTNUNLJCVIT7EKXNMQ, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.216662281 (291h21m25.768132742s) [noise=true , address=89.149.201.147:4011 , piece_id=ZATNMD5HOK35AWGJJ5V66VDJ4YGUVZQJ2OBERQL32ZV5GW2HBSMQ, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.218008314 (291h21m25.769478775s) [noise=true , address=81.171.14.70:4032 , piece_id=CFH2RTR3BLBUB5ZIZCFC6UHT3FHT3B2XSCHQMKXNGDJJGPHEMCTQ, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:28.035150976 (291h21m26.586621437s) [upload=256586]: Next returned nil segment Nov 21 11:25:28.035169622 (291h21m26.586640083s) [upload=256586]: Waiting for error group managing segments... Nov 21 11:25:28.416646836 (291h21m26.968117317s) [noise=true , address=185.167.204.206:4064 , piece_id=SRFIWFQLRAGCNIKYLHMGT4QRUL3WH2QPLLIKAJ7DLFJEQUW5DPPA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.417338854 (291h21m26.968809325s) [noise=true , address=185.167.204.205:4033 , piece_id=FEWQVHPIKN3Q6O5AZ6AGJBLUPZ34PBZFB5X3XNSKGPOGI4AIO7JA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.421089406 (291h21m26.972559867s) [noise=true , address=185.167.204.206:4072 , piece_id=GYCO53OWGOMOHK4FHC5GENN4LTQGSZKZKOI5V534RLV3LHLPKC6Q, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.426252887 (291h21m26.977723348s) [noise=true , address=185.167.204.206:4046 , piece_id=BZ57ZA35ITEOA6CAZJGNODK5MUTXEZWDV7E2OGBZOSMS4OQZYAEA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.431720689 (291h21m26.98319115s) [noise=true , address=185.167.204.205:4027 , piece_id=IVPONKBF2PIHZQAW3L4YA4HOMQV5BQML6SKKCJLIEGT5SZSQ2AOA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.438243951 (291h21m26.989714412s) [noise=true , address=185.167.204.205:4022 , piece_id=BES3YK6PYQV2AKOWSAGHF7DGGWGWSWPIJU2ICAQ5WBHKZ5VT5L6A, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.443364331 (291h21m26.994834792s) [noise=true , address=185.167.204.206:4056 , piece_id=EAMZIK6AVJL7IMFF5VOLW3SMCGYKSKB2VXJTNUNLJCVIT7EKXNMQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.445594728 (291h21m26.997065179s) [noise=true , address=185.167.204.205:4016 , piece_id=EHQS3546UXRXTL2MGFCSRTPDDPAYXEYLMISPP5AWY337MHX7HFLQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.449325822 (291h21m27.000796283s) [noise=true , address=185.167.204.205:4031 , piece_id=ZEI2YSIQC655R7WS43WADXGFT5RKT5LFSN6IZEQ6SHTDVOV7BZBQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.449761211 (291h21m27.001231672s) [noise=true , address=185.167.204.205:4018 , piece_id=7OSLPWJLKFTIJPZEAFMZOAH6JAAMX6HKROI25FODFYBPNJWSMLWA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.450419845 (291h21m27.001890306s) [noise=true , address=185.167.204.206:4062 , piece_id=UF3S6NAVGS2KQJPGA5WXP7YG4RQO44P445SNPCWXKXMONZAYUTNQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.453201252 (291h21m27.004671713s) [noise=true , address=185.167.204.205:4007 , piece_id=MEBGOPZ6MFMQY4UZARS2LKJKEMNN5YGSLKPJGTWSKS45CUNXSRGQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:29.661209626 (291h21m28.212680087s) [noise=true , address=89.149.201.147:4016 , piece_id=TGRWGGGBKDANC7YKROWPRMIYNAVFEVE3RXOSUOINFI7KTRS2IYUQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:29.693169182 (291h21m28.244639643s) [noise=true , address=89.149.201.147:4002 , piece_id=UIYJSCMY3LCLRHZUJB6JFJHHXKMYTVKERTZW7F7X44MWJETPY6XA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:29.721204497 (291h21m28.272674958s) [noise=true , address=89.149.201.147:4008 , piece_id=HCX7I4ICPEHDWTYDFIQMKZ5EE5CGPQLYAIHF7WZMTUUU6CNZPHTA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:29.764584711 (291h21m28.316055172s) [noise=true , address=89.149.201.147:4001 , piece_id=QUS6NB3NQBLJFB6YCK3EBDPKCTWEWGKAT3NOR24TVI65SQKHWDOQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:29.768284075 (291h21m28.319754536s) [noise=true , address=89.149.201.147:4011 , piece_id=ZATNMD5HOK35AWGJJ5V66VDJ4YGUVZQJ2OBERQL32ZV5GW2HBSMQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:29.789681961 (291h21m28.341152422s) [noise=true , address=89.149.201.147:4003 , piece_id=RUAYGN2RGF5JJTWFEGIHWH3BHEUDSS5DN2QJ6YLSHPF5BGUXD3RA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:37.298598682 (291h21m35.850069143s) [noise=true , address=81.171.14.70:4043 , piece_id=T3NF6JR3VJIGDMPNUBLMJ3AOTQP5IHOGJYORMCSS7ZMOENQETZ7A, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:37.497478356 (291h21m36.048948818s) [noise=true , address=81.171.14.70:4032 , piece_id=CFH2RTR3BLBUB5ZIZCFC6UHT3FHT3B2XSCHQMKXNGDJJGPHEMCTQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:40.406166933 (291h21m38.957637394s) [noise=true , address=81.171.14.70:4045 , piece_id=BRWQ7G3E547PJE2RLGSESIB3HMP5C4M5NSKAJNGWSFXUBEO6SYEQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:42.511041157 (291h21m41.062511608s) [noise=true , address=81.171.14.70:4029 , piece_id=DKJM5E7EY7LOSBSEBQOZRC6CI5ZPPE47HFOFQJROPCUMUV567CAA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:26:12.012682521 (291h22m10.564152982s) [noise=true , address=81.171.14.70:4042 , piece_id=XNREZEUZ5L5KZ3L7ZOL6IRK7IHU6DJN4RLEQZRFMDKDH2JWC24CQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:26:15.596046283 (291h22m14.147516744s) [noise=true , address=81.171.14.70:4033 , piece_id=JRKPD4EP6FZM5HISOIDJYZY7FAWWQCADAS47FGW2JM5A7RFJR2BQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:26:18.453059659 (291h22m17.00453012s) [noise=true , address=81.171.14.70:4046 , piece_id=OSONRSF6KJS7F2QJVCC3FPZK4AV7QPG5YSM47CW5LIW7EME44AWA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:26:23.442583344 (291h22m21.994053805s) [noise=true , address=81.171.14.70:4034 , piece_id=ITRLKYVNFSYBLABNGNFO3HRJNMX3MWB345FIMWSIPBNAVZPA47UA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:26:23.442621967 (291h22m21.994092428s) [seg_pos={0 0}, upload=256586]: Segment reached optimal threshold of 26 pieces. Nov 21 11:26:23.442836976 (291h22m21.994307437s) [noise=true , address=81.171.14.70:4044 , piece_id=QYUBNGS2FAFHIWMW7GAGYJ3UVIWVP62LIDCOK6QUXEIQNIXRJDAQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1MUVGecysnTSFcJoHUL7QMWb3h6dtXgUd7FbA672fQCyEWhePr): protocol: expected piece hash; context canceled; EOF Nov 21 11:26:23.442845934 (291h22m21.994316395s) [noise=true , address=81.171.14.70:4044 , piece_id=QYUBNGS2FAFHIWMW7GAGYJ3UVIWVP62LIDCOK6QUXEIQNIXRJDAQ, seg_pos={0 0}, upload=256586]: Long tail cancellation due to optimal threshold Nov 21 11:26:23.443227491 (291h22m21.994697942s) [noise=true , address=81.171.14.70:4027 , piece_id=4FDFCCWF3EIROQAFLOHGS743523OYGSXWQSPQEVD3AMQ6WKDQI7Q, seg_pos={0 0}, upload=256586]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122cYvgTxSvC2aEXnBkRk2o8ARTBXj9gqPoa12vW8hogSN6kpA4): protocol: expected piece hash; context canceled; EOF Nov 21 11:26:23.443237770 (291h22m21.994708231s) [noise=true , address=81.171.14.70:4027 , piece_id=4FDFCCWF3EIROQAFLOHGS743523OYGSXWQSPQEVD3AMQ6WKDQI7Q, seg_pos={0 0}, upload=256586]: Long tail cancellation due to optimal threshold Nov 21 11:26:23.443274720 (291h22m21.994745181s) [seg_pos={0 0}, upload=256586]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Nov 21 11:26:23.443378929 (291h22m21.99484939s) [upload=256586]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 21 11:26:23.443387044 (291h22m21.994857495s) [upload=256586]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 21 11:26:23.685139626 (291h22m22.236610077s) [upload=256586]: Done uploading segments... Nov 21 11:26:23.685160676 (291h22m22.236631127s) [upload=256586]: Upload finished. err: <nil> Duration: 7.271618511s core uplink eu-select-1 eu1-placement-15
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 43 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.................] duration: 3.539947932s trace: https://jaeger.dev.storj.io/trace/6af15a3a67b655d5 FAIL (2025-11-21 11:50:35 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 3.53997371s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 16 out of 60 is failed (threshold=3) [...X.X......X.XX...X.X.X....X............X....X.X.X..X.X..X.] duration: 6.375072317s trace: https://jaeger.dev.storj.io/trace/34d009b2527f9379 FAIL (2025-11-21 11:50:21 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB, Piece ID: N7NO3DOXYOQDEAGQ43OPWR3E5CBYZX5J4A4EJV2VLGKOKJBXCIQQ) context deadline exceeded; piecestore: (Node ID: 1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ, Piece ID: HLV5EPKQNPRWQKHNJLB5ATKRWSS6XW3HMWYXUJL4L7VM7VL4MBZQ) context deadline exceeded; piecestore: (Node ID: 1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, Piece ID: T3KCZD5JV6FAEG6VHBK6HG2E2C6XMP4A6NVAWFWKZTWJIA57L4AA) context deadline exceeded; piecestore: (Node ID: 155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, Piece ID: CECM54T2XTGYW4BFN4TZY2DYUH47J6YO6PGF2QQ6SOCUMLSW6XMQ) context deadline exceeded ======LOGS====== ments... Nov 21 11:49:16.994612805 (291h45m15.546083266s) [upload=256935]: Got next segment. Inline: false Nov 21 11:49:16.994647011 (291h45m15.546117472s) [upload=256935]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 21 11:49:16.994651759 (291h45m15.54612221s) [upload=256935]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 21 11:49:17.243148153 (291h45m15.794618604s) [seg_pos={0 0}, upload=256935]: Begin upload segment... Nov 21 11:49:17.243197467 (291h45m15.794667929s) [seg_pos={0 0}, upload=256935]: Done begin upload segment. Nov 21 11:49:17.243223908 (291h45m15.794694369s) [noise=true , address=77.90.31.11:4555 , piece_id=HLV5EPKQNPRWQKHNJLB5ATKRWSS6XW3HMWYXUJL4L7VM7VL4MBZQ, seg_pos={0 0}, upload=256935]: Uploading piece... Nov 21 11:49:17.243288660 (291h45m15.794759112s) [noise=true , address=145.79.227.11:4731 , piece_id=2NO5DRKFPM22XIWMQU4GDYHKXWYQX2QRAYP3YQBSIWTLAMW6LAPA, seg_pos={0 0}, upload=256935]: Uploading piece... Nov 21 11:49:17.243999364 (291h45m15.795469815s) [noise=true , address=45.91.116.14:4191 , piece_id=3TMBBNLJQ4OZ7RVYS7TYUILMTF5R7TQXXH5RRHDLFQMQAKM66WLA, seg_pos={0 0}, upload=256935]: Uploading piece... Nov 21 11:49:17.244784930 (291h45m15.796255391s) [noise=true , address=145.79.227.10:4403 , piece_id=XS5LV3OLSHZWHVBGVPFV4TFGBPZGHMLGNF6OGVWWQQMTXVWC5QCQ, seg_pos={0 0}, upload=256935]: Uploading piece... Nov 21 11:49:17.245522555 (291h45m15.796993016s) [noise=true , address=77.90.31.13:4628 , piece_id=T3KCZD5JV6FAEG6VHBK6HG2E2C6XMP4A6NVAWFWKZTWJIA57L4AA, seg_pos={0 0}, upload=256935]: Uploading piece... Nov 21 11:49:17.245623898 (291h45m15.797094359s) [noise=true , address=145.79.227.8:4330 , piece_id=ZCJFYOK2Q2B6DDNNWGW6TXFPUJMPOGLV6HAPQO2HEVCGXAQVGBRA, seg_pos={0 0}, upload=256935]: Uploading piece... Nov 21 11:49:17.246035040 (291h45m15.797505491s) [noise=true , address=77.90.31.15:4698 , piece_id=QOPKTWSMOAHGWQ3XVTFJIBMWTJTZ3PJLI6U7BF4TDWR3ZTM2RG5Q, seg_pos={0 0}, upload=256935]: Uploading piece... Nov 21 11:49:17.246184175 (291h45m15.797654636s) [noise=true , address=45.91.116.4:4937 , piece_id=OKQ3DD3KZ73K3RRAE4RR6EG6HOYWZYGGJPKN2QTVN3BWWNKK7PDQ, seg_pos={0 0}, upload=256935]: Uploading piece... Nov 21 11:49:17.246797051 (291h45m15.798267522s) [noise=true , address=45.91.116.10:4081 , piece_id=CSMEKQE6SUSZ52HLGWVDXINNPRIYR6LJM5FBOI5RNFVH7VF7OLBQ, seg_pos={0 0}, upload=256935]: Uploading piece... Nov 21 11:49:17.247334054 (291h45m15.798804515s) [noise=true , address=77.90.31.14:4663 , piece_id=CECM54T2XTGYW4BFN4TZY2DYUH47J6YO6PGF2QQ6SOCUMLSW6XMQ, seg_pos={0 0}, upload=256935]: Uploading piece... Nov 21 11:49:17.247548422 (291h45m15.799018873s) [noise=true , address=145.79.227.30:4761 , piece_id=SDNZAHFY6GR7I4L6BTFGXSAKA5O5MHJ54NHMA77VTAXTCWQQSFQQ, seg_pos={0 0}, upload=256935]: Uploading piece... Nov 21 11:49:17.249260893 (291h45m15.800731354s) [noise=true , address=145.79.227.23:4546 , piece_id=HIT5PPIDXBOIXNY6X3UPCJULOM7YX7BI4VST4IEP6VO7NPK7YMDQ, seg_pos={0 0}, upload=256935]: Uploading piece... Nov 21 11:49:17.249559843 (291h45m15.801030304s) [noise=true , address=145.79.227.28:4726 , piece_id=7XZ277AZOYL6KFSIAAHNTWNZ2PBWMT2YIQFRWNTYEIIINQ6VTRUQ, seg_pos={0 0}, upload=256935]: Uploading piece... Nov 21 11:49:17.251617881 (291h45m15.803088333s) [noise=true , address=145.79.227.31:4798 , piece_id=VKGEFEBHEEU5VUVIVEV7OVY3SUPUXFVGNP7OWNJSBOQAKDYQFQGQ, seg_pos={0 0}, upload=256935]: Uploading piece... Nov 21 11:49:17.252868232 (291h45m15.804338704s) [noise=true , address=145.79.227.8:4331 , piece_id=GK4K577UUQ7IM2NNN5Q4BCJ26JK3EET6VNEEWOPQO7I5VCLOGNFQ, seg_pos={0 0}, upload=256935]: Uploading piece... Nov 21 11:49:17.254615760 (291h45m15.806086221s) [noise=true , address=45.91.116.8:4048 , piece_id=N7NO3DOXYOQDEAGQ43OPWR3E5CBYZX5J4A4EJV2VLGKOKJBXCIQQ, seg_pos={0 0}, upload=256935]: Uploading piece... Nov 21 11:49:18.965747830 (291h45m17.517218281s) [upload=256935]: Next returned nil segment Nov 21 11:49:18.965769751 (291h45m17.517240243s) [upload=256935]: Waiting for error group managing segments... Nov 21 11:49:19.250399321 (291h45m17.801869792s) [noise=true , address=145.79.227.30:4761 , piece_id=SDNZAHFY6GR7I4L6BTFGXSAKA5O5MHJ54NHMA77VTAXTCWQQSFQQ, seg_pos={0 0}, upload=256935]: Done uploading piece. err: <nil> Nov 21 11:49:19.250854327 (291h45m17.802324788s) [noise=true , address=145.79.227.10:4403 , piece_id=XS5LV3OLSHZWHVBGVPFV4TFGBPZGHMLGNF6OGVWWQQMTXVWC5QCQ, seg_pos={0 0}, upload=256935]: Done uploading piece. err: <nil> Nov 21 11:49:19.260959880 (291h45m17.812430341s) [noise=true , address=145.79.227.11:4731 , piece_id=2NO5DRKFPM22XIWMQU4GDYHKXWYQX2QRAYP3YQBSIWTLAMW6LAPA, seg_pos={0 0}, upload=256935]: Done uploading piece. err: <nil> Nov 21 11:49:19.262082247 (291h45m17.813552709s) [noise=true , address=145.79.227.8:4330 , piece_id=ZCJFYOK2Q2B6DDNNWGW6TXFPUJMPOGLV6HAPQO2HEVCGXAQVGBRA, seg_pos={0 0}, upload=256935]: Done uploading piece. err: <nil> Nov 21 11:49:19.268139792 (291h45m17.819610263s) [noise=true , address=145.79.227.28:4726 , piece_id=7XZ277AZOYL6KFSIAAHNTWNZ2PBWMT2YIQFRWNTYEIIINQ6VTRUQ, seg_pos={0 0}, upload=256935]: Done uploading piece. err: <nil> Nov 21 11:49:19.280973680 (291h45m17.832444141s) [noise=true , address=145.79.227.31:4798 , piece_id=VKGEFEBHEEU5VUVIVEV7OVY3SUPUXFVGNP7OWNJSBOQAKDYQFQGQ, seg_pos={0 0}, upload=256935]: Done uploading piece. err: <nil> Nov 21 11:49:19.282868778 (291h45m17.83433924s) [noise=true , address=145.79.227.23:4546 , piece_id=HIT5PPIDXBOIXNY6X3UPCJULOM7YX7BI4VST4IEP6VO7NPK7YMDQ, seg_pos={0 0}, upload=256935]: Done uploading piece. err: <nil> Nov 21 11:49:19.286078351 (291h45m17.837548812s) [noise=true , address=145.79.227.8:4331 , piece_id=GK4K577UUQ7IM2NNN5Q4BCJ26JK3EET6VNEEWOPQO7I5VCLOGNFQ, seg_pos={0 0}, upload=256935]: Done uploading piece. err: <nil> Nov 21 11:49:19.821632116 (291h45m18.373102577s) [noise=true , address=77.90.31.13:4628 , piece_id=T3KCZD5JV6FAEG6VHBK6HG2E2C6XMP4A6NVAWFWKZTWJIA57L4AA, seg_pos={0 0}, upload=256935]: Done uploading piece. err: <nil> Nov 21 11:49:19.824423853 (291h45m18.375894304s) [noise=true , address=77.90.31.14:4663 , piece_id=CECM54T2XTGYW4BFN4TZY2DYUH47J6YO6PGF2QQ6SOCUMLSW6XMQ, seg_pos={0 0}, upload=256935]: Done uploading piece. err: <nil> Nov 21 11:49:19.850707311 (291h45m18.402177772s) [noise=true , address=45.91.116.8:4048 , piece_id=N7NO3DOXYOQDEAGQ43OPWR3E5CBYZX5J4A4EJV2VLGKOKJBXCIQQ, seg_pos={0 0}, upload=256935]: Done uploading piece. err: <nil> Nov 21 11:49:19.908423644 (291h45m18.459894105s) [noise=true , address=77.90.31.11:4555 , piece_id=HLV5EPKQNPRWQKHNJLB5ATKRWSS6XW3HMWYXUJL4L7VM7VL4MBZQ, seg_pos={0 0}, upload=256935]: Done uploading piece. err: <nil> Nov 21 11:49:19.958255192 (291h45m18.509725653s) [noise=true , address=45.91.116.14:4191 , piece_id=3TMBBNLJQ4OZ7RVYS7TYUILMTF5R7TQXXH5RRHDLFQMQAKM66WLA, seg_pos={0 0}, upload=256935]: Done uploading piece. err: <nil> Nov 21 11:49:19.958317902 (291h45m18.509788353s) [seg_pos={0 0}, upload=256935]: Segment reached optimal threshold of 13 pieces. Nov 21 11:49:19.958597444 (291h45m18.510067905s) [noise=true , address=77.90.31.15:4698 , piece_id=QOPKTWSMOAHGWQ3XVTFJIBMWTJTZ3PJLI6U7BF4TDWR3ZTM2RG5Q, seg_pos={0 0}, upload=256935]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj): protocol: write tcp 188.166.225.170:49482->77.90.31.15:4698: use of closed network connection; write tcp 188.166.225.170:49482->77.90.31.15:4698: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49482->77.90.31.15:4698: use of closed network connection Nov 21 11:49:19.958607714 (291h45m18.510078175s) [noise=true , address=77.90.31.15:4698 , piece_id=QOPKTWSMOAHGWQ3XVTFJIBMWTJTZ3PJLI6U7BF4TDWR3ZTM2RG5Q, seg_pos={0 0}, upload=256935]: Long tail cancellation due to optimal threshold Nov 21 11:49:19.958655214 (291h45m18.510125675s) [noise=true , address=45.91.116.10:4081 , piece_id=CSMEKQE6SUSZ52HLGWVDXINNPRIYR6LJM5FBOI5RNFVH7VF7OLBQ, seg_pos={0 0}, upload=256935]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG): protocol: write tcp 188.166.225.170:57522->45.91.116.10:4081: use of closed network connection; write tcp 188.166.225.170:57522->45.91.116.10:4081: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57522->45.91.116.10:4081: use of closed network connection Nov 21 11:49:19.958672887 (291h45m18.510143349s) [noise=true , address=45.91.116.10:4081 , piece_id=CSMEKQE6SUSZ52HLGWVDXINNPRIYR6LJM5FBOI5RNFVH7VF7OLBQ, seg_pos={0 0}, upload=256935]: Long tail cancellation due to optimal threshold Nov 21 11:49:19.958915429 (291h45m18.5103859s) [noise=true , address=45.91.116.4:4937 , piece_id=OKQ3DD3KZ73K3RRAE4RR6EG6HOYWZYGGJPKN2QTVN3BWWNKK7PDQ, seg_pos={0 0}, upload=256935]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq): protocol: expected piece hash; context canceled; EOF Nov 21 11:49:19.958928194 (291h45m18.510398655s) [noise=true , address=45.91.116.4:4937 , piece_id=OKQ3DD3KZ73K3RRAE4RR6EG6HOYWZYGGJPKN2QTVN3BWWNKK7PDQ, seg_pos={0 0}, upload=256935]: Long tail cancellation due to optimal threshold Nov 21 11:49:19.958967489 (291h45m18.51043795s) [seg_pos={0 0}, upload=256935]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 21 11:49:19.959064374 (291h45m18.510534835s) [upload=256935]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 21 11:49:19.959071227 (291h45m18.510541688s) [upload=256935]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 21 11:49:20.260011595 (291h45m18.811482076s) [upload=256935]: Done uploading segments... Nov 21 11:49:20.260040470 (291h45m18.811510931s) [upload=256935]: Upload finished. err: <nil> Duration: 6.375106081s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 44 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX................] duration: 3.347371712s trace: https://jaeger.dev.storj.io/trace/6af15a3a67b655d5 FAIL (2025-11-21 11:50:13 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 3.347397792s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 17 out of 60 is failed (threshold=3) [X...X.X......X.XX...X.X.X....X............X....X.X.X..X.X..X] duration: 30.003141561s trace: https://jaeger.dev.storj.io/trace/34d009b2527f9379 FAIL (2025-11-21 11:49:46 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB, Piece ID: N7NO3DOXYOQDEAGQ43OPWR3E5CBYZX5J4A4EJV2VLGKOKJBXCIQQ) context deadline exceeded; piecestore: (Node ID: 1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ, Piece ID: HLV5EPKQNPRWQKHNJLB5ATKRWSS6XW3HMWYXUJL4L7VM7VL4MBZQ) context deadline exceeded; piecestore: (Node ID: 1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, Piece ID: T3KCZD5JV6FAEG6VHBK6HG2E2C6XMP4A6NVAWFWKZTWJIA57L4AA) context deadline exceeded; piecestore: (Node ID: 155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, Piece ID: CECM54T2XTGYW4BFN4TZY2DYUH47J6YO6PGF2QQ6SOCUMLSW6XMQ) context deadline exceeded ======LOGS====== ments... Nov 21 11:49:16.994612805 (291h45m15.546083266s) [upload=256935]: Got next segment. Inline: false Nov 21 11:49:16.994647011 (291h45m15.546117472s) [upload=256935]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 21 11:49:16.994651759 (291h45m15.54612221s) [upload=256935]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 21 11:49:17.243148153 (291h45m15.794618604s) [seg_pos={0 0}, upload=256935]: Begin upload segment... Nov 21 11:49:17.243197467 (291h45m15.794667929s) [seg_pos={0 0}, upload=256935]: Done begin upload segment. Nov 21 11:49:17.243223908 (291h45m15.794694369s) [noise=true , address=77.90.31.11:4555 , piece_id=HLV5EPKQNPRWQKHNJLB5ATKRWSS6XW3HMWYXUJL4L7VM7VL4MBZQ, seg_pos={0 0}, upload=256935]: Uploading piece... Nov 21 11:49:17.243288660 (291h45m15.794759112s) [noise=true , address=145.79.227.11:4731 , piece_id=2NO5DRKFPM22XIWMQU4GDYHKXWYQX2QRAYP3YQBSIWTLAMW6LAPA, seg_pos={0 0}, upload=256935]: Uploading piece... Nov 21 11:49:17.243999364 (291h45m15.795469815s) [noise=true , address=45.91.116.14:4191 , piece_id=3TMBBNLJQ4OZ7RVYS7TYUILMTF5R7TQXXH5RRHDLFQMQAKM66WLA, seg_pos={0 0}, upload=256935]: Uploading piece... Nov 21 11:49:17.244784930 (291h45m15.796255391s) [noise=true , address=145.79.227.10:4403 , piece_id=XS5LV3OLSHZWHVBGVPFV4TFGBPZGHMLGNF6OGVWWQQMTXVWC5QCQ, seg_pos={0 0}, upload=256935]: Uploading piece... Nov 21 11:49:17.245522555 (291h45m15.796993016s) [noise=true , address=77.90.31.13:4628 , piece_id=T3KCZD5JV6FAEG6VHBK6HG2E2C6XMP4A6NVAWFWKZTWJIA57L4AA, seg_pos={0 0}, upload=256935]: Uploading piece... Nov 21 11:49:17.245623898 (291h45m15.797094359s) [noise=true , address=145.79.227.8:4330 , piece_id=ZCJFYOK2Q2B6DDNNWGW6TXFPUJMPOGLV6HAPQO2HEVCGXAQVGBRA, seg_pos={0 0}, upload=256935]: Uploading piece... Nov 21 11:49:17.246035040 (291h45m15.797505491s) [noise=true , address=77.90.31.15:4698 , piece_id=QOPKTWSMOAHGWQ3XVTFJIBMWTJTZ3PJLI6U7BF4TDWR3ZTM2RG5Q, seg_pos={0 0}, upload=256935]: Uploading piece... Nov 21 11:49:17.246184175 (291h45m15.797654636s) [noise=true , address=45.91.116.4:4937 , piece_id=OKQ3DD3KZ73K3RRAE4RR6EG6HOYWZYGGJPKN2QTVN3BWWNKK7PDQ, seg_pos={0 0}, upload=256935]: Uploading piece... Nov 21 11:49:17.246797051 (291h45m15.798267522s) [noise=true , address=45.91.116.10:4081 , piece_id=CSMEKQE6SUSZ52HLGWVDXINNPRIYR6LJM5FBOI5RNFVH7VF7OLBQ, seg_pos={0 0}, upload=256935]: Uploading piece... Nov 21 11:49:17.247334054 (291h45m15.798804515s) [noise=true , address=77.90.31.14:4663 , piece_id=CECM54T2XTGYW4BFN4TZY2DYUH47J6YO6PGF2QQ6SOCUMLSW6XMQ, seg_pos={0 0}, upload=256935]: Uploading piece... Nov 21 11:49:17.247548422 (291h45m15.799018873s) [noise=true , address=145.79.227.30:4761 , piece_id=SDNZAHFY6GR7I4L6BTFGXSAKA5O5MHJ54NHMA77VTAXTCWQQSFQQ, seg_pos={0 0}, upload=256935]: Uploading piece... Nov 21 11:49:17.249260893 (291h45m15.800731354s) [noise=true , address=145.79.227.23:4546 , piece_id=HIT5PPIDXBOIXNY6X3UPCJULOM7YX7BI4VST4IEP6VO7NPK7YMDQ, seg_pos={0 0}, upload=256935]: Uploading piece... Nov 21 11:49:17.249559843 (291h45m15.801030304s) [noise=true , address=145.79.227.28:4726 , piece_id=7XZ277AZOYL6KFSIAAHNTWNZ2PBWMT2YIQFRWNTYEIIINQ6VTRUQ, seg_pos={0 0}, upload=256935]: Uploading piece... Nov 21 11:49:17.251617881 (291h45m15.803088333s) [noise=true , address=145.79.227.31:4798 , piece_id=VKGEFEBHEEU5VUVIVEV7OVY3SUPUXFVGNP7OWNJSBOQAKDYQFQGQ, seg_pos={0 0}, upload=256935]: Uploading piece... Nov 21 11:49:17.252868232 (291h45m15.804338704s) [noise=true , address=145.79.227.8:4331 , piece_id=GK4K577UUQ7IM2NNN5Q4BCJ26JK3EET6VNEEWOPQO7I5VCLOGNFQ, seg_pos={0 0}, upload=256935]: Uploading piece... Nov 21 11:49:17.254615760 (291h45m15.806086221s) [noise=true , address=45.91.116.8:4048 , piece_id=N7NO3DOXYOQDEAGQ43OPWR3E5CBYZX5J4A4EJV2VLGKOKJBXCIQQ, seg_pos={0 0}, upload=256935]: Uploading piece... Nov 21 11:49:18.965747830 (291h45m17.517218281s) [upload=256935]: Next returned nil segment Nov 21 11:49:18.965769751 (291h45m17.517240243s) [upload=256935]: Waiting for error group managing segments... Nov 21 11:49:19.250399321 (291h45m17.801869792s) [noise=true , address=145.79.227.30:4761 , piece_id=SDNZAHFY6GR7I4L6BTFGXSAKA5O5MHJ54NHMA77VTAXTCWQQSFQQ, seg_pos={0 0}, upload=256935]: Done uploading piece. err: <nil> Nov 21 11:49:19.250854327 (291h45m17.802324788s) [noise=true , address=145.79.227.10:4403 , piece_id=XS5LV3OLSHZWHVBGVPFV4TFGBPZGHMLGNF6OGVWWQQMTXVWC5QCQ, seg_pos={0 0}, upload=256935]: Done uploading piece. err: <nil> Nov 21 11:49:19.260959880 (291h45m17.812430341s) [noise=true , address=145.79.227.11:4731 , piece_id=2NO5DRKFPM22XIWMQU4GDYHKXWYQX2QRAYP3YQBSIWTLAMW6LAPA, seg_pos={0 0}, upload=256935]: Done uploading piece. err: <nil> Nov 21 11:49:19.262082247 (291h45m17.813552709s) [noise=true , address=145.79.227.8:4330 , piece_id=ZCJFYOK2Q2B6DDNNWGW6TXFPUJMPOGLV6HAPQO2HEVCGXAQVGBRA, seg_pos={0 0}, upload=256935]: Done uploading piece. err: <nil> Nov 21 11:49:19.268139792 (291h45m17.819610263s) [noise=true , address=145.79.227.28:4726 , piece_id=7XZ277AZOYL6KFSIAAHNTWNZ2PBWMT2YIQFRWNTYEIIINQ6VTRUQ, seg_pos={0 0}, upload=256935]: Done uploading piece. err: <nil> Nov 21 11:49:19.280973680 (291h45m17.832444141s) [noise=true , address=145.79.227.31:4798 , piece_id=VKGEFEBHEEU5VUVIVEV7OVY3SUPUXFVGNP7OWNJSBOQAKDYQFQGQ, seg_pos={0 0}, upload=256935]: Done uploading piece. err: <nil> Nov 21 11:49:19.282868778 (291h45m17.83433924s) [noise=true , address=145.79.227.23:4546 , piece_id=HIT5PPIDXBOIXNY6X3UPCJULOM7YX7BI4VST4IEP6VO7NPK7YMDQ, seg_pos={0 0}, upload=256935]: Done uploading piece. err: <nil> Nov 21 11:49:19.286078351 (291h45m17.837548812s) [noise=true , address=145.79.227.8:4331 , piece_id=GK4K577UUQ7IM2NNN5Q4BCJ26JK3EET6VNEEWOPQO7I5VCLOGNFQ, seg_pos={0 0}, upload=256935]: Done uploading piece. err: <nil> Nov 21 11:49:19.821632116 (291h45m18.373102577s) [noise=true , address=77.90.31.13:4628 , piece_id=T3KCZD5JV6FAEG6VHBK6HG2E2C6XMP4A6NVAWFWKZTWJIA57L4AA, seg_pos={0 0}, upload=256935]: Done uploading piece. err: <nil> Nov 21 11:49:19.824423853 (291h45m18.375894304s) [noise=true , address=77.90.31.14:4663 , piece_id=CECM54T2XTGYW4BFN4TZY2DYUH47J6YO6PGF2QQ6SOCUMLSW6XMQ, seg_pos={0 0}, upload=256935]: Done uploading piece. err: <nil> Nov 21 11:49:19.850707311 (291h45m18.402177772s) [noise=true , address=45.91.116.8:4048 , piece_id=N7NO3DOXYOQDEAGQ43OPWR3E5CBYZX5J4A4EJV2VLGKOKJBXCIQQ, seg_pos={0 0}, upload=256935]: Done uploading piece. err: <nil> Nov 21 11:49:19.908423644 (291h45m18.459894105s) [noise=true , address=77.90.31.11:4555 , piece_id=HLV5EPKQNPRWQKHNJLB5ATKRWSS6XW3HMWYXUJL4L7VM7VL4MBZQ, seg_pos={0 0}, upload=256935]: Done uploading piece. err: <nil> Nov 21 11:49:19.958255192 (291h45m18.509725653s) [noise=true , address=45.91.116.14:4191 , piece_id=3TMBBNLJQ4OZ7RVYS7TYUILMTF5R7TQXXH5RRHDLFQMQAKM66WLA, seg_pos={0 0}, upload=256935]: Done uploading piece. err: <nil> Nov 21 11:49:19.958317902 (291h45m18.509788353s) [seg_pos={0 0}, upload=256935]: Segment reached optimal threshold of 13 pieces. Nov 21 11:49:19.958597444 (291h45m18.510067905s) [noise=true , address=77.90.31.15:4698 , piece_id=QOPKTWSMOAHGWQ3XVTFJIBMWTJTZ3PJLI6U7BF4TDWR3ZTM2RG5Q, seg_pos={0 0}, upload=256935]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj): protocol: write tcp 188.166.225.170:49482->77.90.31.15:4698: use of closed network connection; write tcp 188.166.225.170:49482->77.90.31.15:4698: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49482->77.90.31.15:4698: use of closed network connection Nov 21 11:49:19.958607714 (291h45m18.510078175s) [noise=true , address=77.90.31.15:4698 , piece_id=QOPKTWSMOAHGWQ3XVTFJIBMWTJTZ3PJLI6U7BF4TDWR3ZTM2RG5Q, seg_pos={0 0}, upload=256935]: Long tail cancellation due to optimal threshold Nov 21 11:49:19.958655214 (291h45m18.510125675s) [noise=true , address=45.91.116.10:4081 , piece_id=CSMEKQE6SUSZ52HLGWVDXINNPRIYR6LJM5FBOI5RNFVH7VF7OLBQ, seg_pos={0 0}, upload=256935]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG): protocol: write tcp 188.166.225.170:57522->45.91.116.10:4081: use of closed network connection; write tcp 188.166.225.170:57522->45.91.116.10:4081: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57522->45.91.116.10:4081: use of closed network connection Nov 21 11:49:19.958672887 (291h45m18.510143349s) [noise=true , address=45.91.116.10:4081 , piece_id=CSMEKQE6SUSZ52HLGWVDXINNPRIYR6LJM5FBOI5RNFVH7VF7OLBQ, seg_pos={0 0}, upload=256935]: Long tail cancellation due to optimal threshold Nov 21 11:49:19.958915429 (291h45m18.5103859s) [noise=true , address=45.91.116.4:4937 , piece_id=OKQ3DD3KZ73K3RRAE4RR6EG6HOYWZYGGJPKN2QTVN3BWWNKK7PDQ, seg_pos={0 0}, upload=256935]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq): protocol: expected piece hash; context canceled; EOF Nov 21 11:49:19.958928194 (291h45m18.510398655s) [noise=true , address=45.91.116.4:4937 , piece_id=OKQ3DD3KZ73K3RRAE4RR6EG6HOYWZYGGJPKN2QTVN3BWWNKK7PDQ, seg_pos={0 0}, upload=256935]: Long tail cancellation due to optimal threshold Nov 21 11:49:19.958967489 (291h45m18.51043795s) [seg_pos={0 0}, upload=256935]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 21 11:49:19.959064374 (291h45m18.510534835s) [upload=256935]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 21 11:49:19.959071227 (291h45m18.510541688s) [upload=256935]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 21 11:49:20.260011595 (291h45m18.811482076s) [upload=256935]: Done uploading segments... Nov 21 11:49:20.260040470 (291h45m18.811510931s) [upload=256935]: Upload finished. err: <nil> Duration: 30.003166049s core uplink us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 2 out of 60 is failed (threshold=3) [.....................X.X....................................] duration: 5.53975258s trace: https://jaeger.dev.storj.io/trace/a7f69ee106efb7f FAIL (2025-11-21 11:49:30 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== upload=256586]: Uploading piece... Nov 21 11:25:27.210346024 (291h21m25.761816485s) [noise=true , address=89.149.201.147:4008 , piece_id=HCX7I4ICPEHDWTYDFIQMKZ5EE5CGPQLYAIHF7WZMTUUU6CNZPHTA, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.211425289 (291h21m25.76289575s) [noise=true , address=81.171.14.70:4044 , piece_id=QYUBNGS2FAFHIWMW7GAGYJ3UVIWVP62LIDCOK6QUXEIQNIXRJDAQ, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.213103795 (291h21m25.764574246s) [noise=true , address=185.167.204.205:4022 , piece_id=BES3YK6PYQV2AKOWSAGHF7DGGWGWSWPIJU2ICAQ5WBHKZ5VT5L6A, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.213408395 (291h21m25.764878856s) [noise=true , address=185.167.204.205:4027 , piece_id=IVPONKBF2PIHZQAW3L4YA4HOMQV5BQML6SKKCJLIEGT5SZSQ2AOA, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.214501747 (291h21m25.765972208s) [noise=true , address=81.171.14.70:4042 , piece_id=XNREZEUZ5L5KZ3L7ZOL6IRK7IHU6DJN4RLEQZRFMDKDH2JWC24CQ, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.215013752 (291h21m25.766484213s) [noise=true , address=81.171.14.70:4034 , piece_id=ITRLKYVNFSYBLABNGNFO3HRJNMX3MWB345FIMWSIPBNAVZPA47UA, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.215087572 (291h21m25.766558044s) [noise=true , address=185.167.204.205:4031 , piece_id=ZEI2YSIQC655R7WS43WADXGFT5RKT5LFSN6IZEQ6SHTDVOV7BZBQ, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.216432844 (291h21m25.767903305s) [noise=true , address=185.167.204.206:4056 , piece_id=EAMZIK6AVJL7IMFF5VOLW3SMCGYKSKB2VXJTNUNLJCVIT7EKXNMQ, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.216662281 (291h21m25.768132742s) [noise=true , address=89.149.201.147:4011 , piece_id=ZATNMD5HOK35AWGJJ5V66VDJ4YGUVZQJ2OBERQL32ZV5GW2HBSMQ, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.218008314 (291h21m25.769478775s) [noise=true , address=81.171.14.70:4032 , piece_id=CFH2RTR3BLBUB5ZIZCFC6UHT3FHT3B2XSCHQMKXNGDJJGPHEMCTQ, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:28.035150976 (291h21m26.586621437s) [upload=256586]: Next returned nil segment Nov 21 11:25:28.035169622 (291h21m26.586640083s) [upload=256586]: Waiting for error group managing segments... Nov 21 11:25:28.416646836 (291h21m26.968117317s) [noise=true , address=185.167.204.206:4064 , piece_id=SRFIWFQLRAGCNIKYLHMGT4QRUL3WH2QPLLIKAJ7DLFJEQUW5DPPA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.417338854 (291h21m26.968809325s) [noise=true , address=185.167.204.205:4033 , piece_id=FEWQVHPIKN3Q6O5AZ6AGJBLUPZ34PBZFB5X3XNSKGPOGI4AIO7JA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.421089406 (291h21m26.972559867s) [noise=true , address=185.167.204.206:4072 , piece_id=GYCO53OWGOMOHK4FHC5GENN4LTQGSZKZKOI5V534RLV3LHLPKC6Q, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.426252887 (291h21m26.977723348s) [noise=true , address=185.167.204.206:4046 , piece_id=BZ57ZA35ITEOA6CAZJGNODK5MUTXEZWDV7E2OGBZOSMS4OQZYAEA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.431720689 (291h21m26.98319115s) [noise=true , address=185.167.204.205:4027 , piece_id=IVPONKBF2PIHZQAW3L4YA4HOMQV5BQML6SKKCJLIEGT5SZSQ2AOA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.438243951 (291h21m26.989714412s) [noise=true , address=185.167.204.205:4022 , piece_id=BES3YK6PYQV2AKOWSAGHF7DGGWGWSWPIJU2ICAQ5WBHKZ5VT5L6A, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.443364331 (291h21m26.994834792s) [noise=true , address=185.167.204.206:4056 , piece_id=EAMZIK6AVJL7IMFF5VOLW3SMCGYKSKB2VXJTNUNLJCVIT7EKXNMQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.445594728 (291h21m26.997065179s) [noise=true , address=185.167.204.205:4016 , piece_id=EHQS3546UXRXTL2MGFCSRTPDDPAYXEYLMISPP5AWY337MHX7HFLQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.449325822 (291h21m27.000796283s) [noise=true , address=185.167.204.205:4031 , piece_id=ZEI2YSIQC655R7WS43WADXGFT5RKT5LFSN6IZEQ6SHTDVOV7BZBQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.449761211 (291h21m27.001231672s) [noise=true , address=185.167.204.205:4018 , piece_id=7OSLPWJLKFTIJPZEAFMZOAH6JAAMX6HKROI25FODFYBPNJWSMLWA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.450419845 (291h21m27.001890306s) [noise=true , address=185.167.204.206:4062 , piece_id=UF3S6NAVGS2KQJPGA5WXP7YG4RQO44P445SNPCWXKXMONZAYUTNQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.453201252 (291h21m27.004671713s) [noise=true , address=185.167.204.205:4007 , piece_id=MEBGOPZ6MFMQY4UZARS2LKJKEMNN5YGSLKPJGTWSKS45CUNXSRGQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:29.661209626 (291h21m28.212680087s) [noise=true , address=89.149.201.147:4016 , piece_id=TGRWGGGBKDANC7YKROWPRMIYNAVFEVE3RXOSUOINFI7KTRS2IYUQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:29.693169182 (291h21m28.244639643s) [noise=true , address=89.149.201.147:4002 , piece_id=UIYJSCMY3LCLRHZUJB6JFJHHXKMYTVKERTZW7F7X44MWJETPY6XA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:29.721204497 (291h21m28.272674958s) [noise=true , address=89.149.201.147:4008 , piece_id=HCX7I4ICPEHDWTYDFIQMKZ5EE5CGPQLYAIHF7WZMTUUU6CNZPHTA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:29.764584711 (291h21m28.316055172s) [noise=true , address=89.149.201.147:4001 , piece_id=QUS6NB3NQBLJFB6YCK3EBDPKCTWEWGKAT3NOR24TVI65SQKHWDOQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:29.768284075 (291h21m28.319754536s) [noise=true , address=89.149.201.147:4011 , piece_id=ZATNMD5HOK35AWGJJ5V66VDJ4YGUVZQJ2OBERQL32ZV5GW2HBSMQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:29.789681961 (291h21m28.341152422s) [noise=true , address=89.149.201.147:4003 , piece_id=RUAYGN2RGF5JJTWFEGIHWH3BHEUDSS5DN2QJ6YLSHPF5BGUXD3RA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:37.298598682 (291h21m35.850069143s) [noise=true , address=81.171.14.70:4043 , piece_id=T3NF6JR3VJIGDMPNUBLMJ3AOTQP5IHOGJYORMCSS7ZMOENQETZ7A, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:37.497478356 (291h21m36.048948818s) [noise=true , address=81.171.14.70:4032 , piece_id=CFH2RTR3BLBUB5ZIZCFC6UHT3FHT3B2XSCHQMKXNGDJJGPHEMCTQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:40.406166933 (291h21m38.957637394s) [noise=true , address=81.171.14.70:4045 , piece_id=BRWQ7G3E547PJE2RLGSESIB3HMP5C4M5NSKAJNGWSFXUBEO6SYEQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:42.511041157 (291h21m41.062511608s) [noise=true , address=81.171.14.70:4029 , piece_id=DKJM5E7EY7LOSBSEBQOZRC6CI5ZPPE47HFOFQJROPCUMUV567CAA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:26:12.012682521 (291h22m10.564152982s) [noise=true , address=81.171.14.70:4042 , piece_id=XNREZEUZ5L5KZ3L7ZOL6IRK7IHU6DJN4RLEQZRFMDKDH2JWC24CQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:26:15.596046283 (291h22m14.147516744s) [noise=true , address=81.171.14.70:4033 , piece_id=JRKPD4EP6FZM5HISOIDJYZY7FAWWQCADAS47FGW2JM5A7RFJR2BQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:26:18.453059659 (291h22m17.00453012s) [noise=true , address=81.171.14.70:4046 , piece_id=OSONRSF6KJS7F2QJVCC3FPZK4AV7QPG5YSM47CW5LIW7EME44AWA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:26:23.442583344 (291h22m21.994053805s) [noise=true , address=81.171.14.70:4034 , piece_id=ITRLKYVNFSYBLABNGNFO3HRJNMX3MWB345FIMWSIPBNAVZPA47UA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:26:23.442621967 (291h22m21.994092428s) [seg_pos={0 0}, upload=256586]: Segment reached optimal threshold of 26 pieces. Nov 21 11:26:23.442836976 (291h22m21.994307437s) [noise=true , address=81.171.14.70:4044 , piece_id=QYUBNGS2FAFHIWMW7GAGYJ3UVIWVP62LIDCOK6QUXEIQNIXRJDAQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1MUVGecysnTSFcJoHUL7QMWb3h6dtXgUd7FbA672fQCyEWhePr): protocol: expected piece hash; context canceled; EOF Nov 21 11:26:23.442845934 (291h22m21.994316395s) [noise=true , address=81.171.14.70:4044 , piece_id=QYUBNGS2FAFHIWMW7GAGYJ3UVIWVP62LIDCOK6QUXEIQNIXRJDAQ, seg_pos={0 0}, upload=256586]: Long tail cancellation due to optimal threshold Nov 21 11:26:23.443227491 (291h22m21.994697942s) [noise=true , address=81.171.14.70:4027 , piece_id=4FDFCCWF3EIROQAFLOHGS743523OYGSXWQSPQEVD3AMQ6WKDQI7Q, seg_pos={0 0}, upload=256586]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122cYvgTxSvC2aEXnBkRk2o8ARTBXj9gqPoa12vW8hogSN6kpA4): protocol: expected piece hash; context canceled; EOF Nov 21 11:26:23.443237770 (291h22m21.994708231s) [noise=true , address=81.171.14.70:4027 , piece_id=4FDFCCWF3EIROQAFLOHGS743523OYGSXWQSPQEVD3AMQ6WKDQI7Q, seg_pos={0 0}, upload=256586]: Long tail cancellation due to optimal threshold Nov 21 11:26:23.443274720 (291h22m21.994745181s) [seg_pos={0 0}, upload=256586]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Nov 21 11:26:23.443378929 (291h22m21.99484939s) [upload=256586]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 21 11:26:23.443387044 (291h22m21.994857495s) [upload=256586]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 21 11:26:23.685139626 (291h22m22.236610077s) [upload=256586]: Done uploading segments... Nov 21 11:26:23.685160676 (291h22m22.236631127s) [upload=256586]: Upload finished. err: <nil> Duration: 5.539782757s core uplink eu-select-1 eu1-placement-15
checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us1.gateway) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 257.494388ms remote: "tcp:207.211.208.129:443" target: https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222 trace: https://jaeger.dev.storj.io/trace/3e5c193367f9a974 FAIL (2025-11-21 11:49:29 UTC) orbiter warning: Get "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222": context deadline exceeded Duration: 257.635517ms edge gateway vhost bgp us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 45 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX...............] duration: 3.447567422s trace: https://jaeger.dev.storj.io/trace/6af15a3a67b655d5 FAIL (2025-11-21 11:49:15 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 3.447622696s core uplink us-select-1 us1-placement-20
checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us1.gateway) antiblip: 1 out of 60 is failed (threshold=3) [.X..........................................................] duration: 264.59026ms remote: "tcp:207.211.208.129:443" target: https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222 trace: https://jaeger.dev.storj.io/trace/3e5c193367f9a974 FAIL (2025-11-21 11:49:05 UTC) orbiter warning: Get "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222": context deadline exceeded Duration: 264.661786ms edge gateway vhost bgp us1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 2 out of 60 is failed (threshold=3) [......................X.X...................................] duration: 5.893265717s trace: https://jaeger.dev.storj.io/trace/a7f69ee106efb7f FAIL (2025-11-21 11:48:51 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== upload=256586]: Uploading piece... Nov 21 11:25:27.210346024 (291h21m25.761816485s) [noise=true , address=89.149.201.147:4008 , piece_id=HCX7I4ICPEHDWTYDFIQMKZ5EE5CGPQLYAIHF7WZMTUUU6CNZPHTA, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.211425289 (291h21m25.76289575s) [noise=true , address=81.171.14.70:4044 , piece_id=QYUBNGS2FAFHIWMW7GAGYJ3UVIWVP62LIDCOK6QUXEIQNIXRJDAQ, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.213103795 (291h21m25.764574246s) [noise=true , address=185.167.204.205:4022 , piece_id=BES3YK6PYQV2AKOWSAGHF7DGGWGWSWPIJU2ICAQ5WBHKZ5VT5L6A, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.213408395 (291h21m25.764878856s) [noise=true , address=185.167.204.205:4027 , piece_id=IVPONKBF2PIHZQAW3L4YA4HOMQV5BQML6SKKCJLIEGT5SZSQ2AOA, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.214501747 (291h21m25.765972208s) [noise=true , address=81.171.14.70:4042 , piece_id=XNREZEUZ5L5KZ3L7ZOL6IRK7IHU6DJN4RLEQZRFMDKDH2JWC24CQ, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.215013752 (291h21m25.766484213s) [noise=true , address=81.171.14.70:4034 , piece_id=ITRLKYVNFSYBLABNGNFO3HRJNMX3MWB345FIMWSIPBNAVZPA47UA, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.215087572 (291h21m25.766558044s) [noise=true , address=185.167.204.205:4031 , piece_id=ZEI2YSIQC655R7WS43WADXGFT5RKT5LFSN6IZEQ6SHTDVOV7BZBQ, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.216432844 (291h21m25.767903305s) [noise=true , address=185.167.204.206:4056 , piece_id=EAMZIK6AVJL7IMFF5VOLW3SMCGYKSKB2VXJTNUNLJCVIT7EKXNMQ, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.216662281 (291h21m25.768132742s) [noise=true , address=89.149.201.147:4011 , piece_id=ZATNMD5HOK35AWGJJ5V66VDJ4YGUVZQJ2OBERQL32ZV5GW2HBSMQ, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.218008314 (291h21m25.769478775s) [noise=true , address=81.171.14.70:4032 , piece_id=CFH2RTR3BLBUB5ZIZCFC6UHT3FHT3B2XSCHQMKXNGDJJGPHEMCTQ, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:28.035150976 (291h21m26.586621437s) [upload=256586]: Next returned nil segment Nov 21 11:25:28.035169622 (291h21m26.586640083s) [upload=256586]: Waiting for error group managing segments... Nov 21 11:25:28.416646836 (291h21m26.968117317s) [noise=true , address=185.167.204.206:4064 , piece_id=SRFIWFQLRAGCNIKYLHMGT4QRUL3WH2QPLLIKAJ7DLFJEQUW5DPPA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.417338854 (291h21m26.968809325s) [noise=true , address=185.167.204.205:4033 , piece_id=FEWQVHPIKN3Q6O5AZ6AGJBLUPZ34PBZFB5X3XNSKGPOGI4AIO7JA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.421089406 (291h21m26.972559867s) [noise=true , address=185.167.204.206:4072 , piece_id=GYCO53OWGOMOHK4FHC5GENN4LTQGSZKZKOI5V534RLV3LHLPKC6Q, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.426252887 (291h21m26.977723348s) [noise=true , address=185.167.204.206:4046 , piece_id=BZ57ZA35ITEOA6CAZJGNODK5MUTXEZWDV7E2OGBZOSMS4OQZYAEA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.431720689 (291h21m26.98319115s) [noise=true , address=185.167.204.205:4027 , piece_id=IVPONKBF2PIHZQAW3L4YA4HOMQV5BQML6SKKCJLIEGT5SZSQ2AOA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.438243951 (291h21m26.989714412s) [noise=true , address=185.167.204.205:4022 , piece_id=BES3YK6PYQV2AKOWSAGHF7DGGWGWSWPIJU2ICAQ5WBHKZ5VT5L6A, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.443364331 (291h21m26.994834792s) [noise=true , address=185.167.204.206:4056 , piece_id=EAMZIK6AVJL7IMFF5VOLW3SMCGYKSKB2VXJTNUNLJCVIT7EKXNMQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.445594728 (291h21m26.997065179s) [noise=true , address=185.167.204.205:4016 , piece_id=EHQS3546UXRXTL2MGFCSRTPDDPAYXEYLMISPP5AWY337MHX7HFLQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.449325822 (291h21m27.000796283s) [noise=true , address=185.167.204.205:4031 , piece_id=ZEI2YSIQC655R7WS43WADXGFT5RKT5LFSN6IZEQ6SHTDVOV7BZBQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.449761211 (291h21m27.001231672s) [noise=true , address=185.167.204.205:4018 , piece_id=7OSLPWJLKFTIJPZEAFMZOAH6JAAMX6HKROI25FODFYBPNJWSMLWA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.450419845 (291h21m27.001890306s) [noise=true , address=185.167.204.206:4062 , piece_id=UF3S6NAVGS2KQJPGA5WXP7YG4RQO44P445SNPCWXKXMONZAYUTNQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.453201252 (291h21m27.004671713s) [noise=true , address=185.167.204.205:4007 , piece_id=MEBGOPZ6MFMQY4UZARS2LKJKEMNN5YGSLKPJGTWSKS45CUNXSRGQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:29.661209626 (291h21m28.212680087s) [noise=true , address=89.149.201.147:4016 , piece_id=TGRWGGGBKDANC7YKROWPRMIYNAVFEVE3RXOSUOINFI7KTRS2IYUQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:29.693169182 (291h21m28.244639643s) [noise=true , address=89.149.201.147:4002 , piece_id=UIYJSCMY3LCLRHZUJB6JFJHHXKMYTVKERTZW7F7X44MWJETPY6XA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:29.721204497 (291h21m28.272674958s) [noise=true , address=89.149.201.147:4008 , piece_id=HCX7I4ICPEHDWTYDFIQMKZ5EE5CGPQLYAIHF7WZMTUUU6CNZPHTA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:29.764584711 (291h21m28.316055172s) [noise=true , address=89.149.201.147:4001 , piece_id=QUS6NB3NQBLJFB6YCK3EBDPKCTWEWGKAT3NOR24TVI65SQKHWDOQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:29.768284075 (291h21m28.319754536s) [noise=true , address=89.149.201.147:4011 , piece_id=ZATNMD5HOK35AWGJJ5V66VDJ4YGUVZQJ2OBERQL32ZV5GW2HBSMQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:29.789681961 (291h21m28.341152422s) [noise=true , address=89.149.201.147:4003 , piece_id=RUAYGN2RGF5JJTWFEGIHWH3BHEUDSS5DN2QJ6YLSHPF5BGUXD3RA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:37.298598682 (291h21m35.850069143s) [noise=true , address=81.171.14.70:4043 , piece_id=T3NF6JR3VJIGDMPNUBLMJ3AOTQP5IHOGJYORMCSS7ZMOENQETZ7A, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:37.497478356 (291h21m36.048948818s) [noise=true , address=81.171.14.70:4032 , piece_id=CFH2RTR3BLBUB5ZIZCFC6UHT3FHT3B2XSCHQMKXNGDJJGPHEMCTQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:40.406166933 (291h21m38.957637394s) [noise=true , address=81.171.14.70:4045 , piece_id=BRWQ7G3E547PJE2RLGSESIB3HMP5C4M5NSKAJNGWSFXUBEO6SYEQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:42.511041157 (291h21m41.062511608s) [noise=true , address=81.171.14.70:4029 , piece_id=DKJM5E7EY7LOSBSEBQOZRC6CI5ZPPE47HFOFQJROPCUMUV567CAA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:26:12.012682521 (291h22m10.564152982s) [noise=true , address=81.171.14.70:4042 , piece_id=XNREZEUZ5L5KZ3L7ZOL6IRK7IHU6DJN4RLEQZRFMDKDH2JWC24CQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:26:15.596046283 (291h22m14.147516744s) [noise=true , address=81.171.14.70:4033 , piece_id=JRKPD4EP6FZM5HISOIDJYZY7FAWWQCADAS47FGW2JM5A7RFJR2BQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:26:18.453059659 (291h22m17.00453012s) [noise=true , address=81.171.14.70:4046 , piece_id=OSONRSF6KJS7F2QJVCC3FPZK4AV7QPG5YSM47CW5LIW7EME44AWA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:26:23.442583344 (291h22m21.994053805s) [noise=true , address=81.171.14.70:4034 , piece_id=ITRLKYVNFSYBLABNGNFO3HRJNMX3MWB345FIMWSIPBNAVZPA47UA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:26:23.442621967 (291h22m21.994092428s) [seg_pos={0 0}, upload=256586]: Segment reached optimal threshold of 26 pieces. Nov 21 11:26:23.442836976 (291h22m21.994307437s) [noise=true , address=81.171.14.70:4044 , piece_id=QYUBNGS2FAFHIWMW7GAGYJ3UVIWVP62LIDCOK6QUXEIQNIXRJDAQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1MUVGecysnTSFcJoHUL7QMWb3h6dtXgUd7FbA672fQCyEWhePr): protocol: expected piece hash; context canceled; EOF Nov 21 11:26:23.442845934 (291h22m21.994316395s) [noise=true , address=81.171.14.70:4044 , piece_id=QYUBNGS2FAFHIWMW7GAGYJ3UVIWVP62LIDCOK6QUXEIQNIXRJDAQ, seg_pos={0 0}, upload=256586]: Long tail cancellation due to optimal threshold Nov 21 11:26:23.443227491 (291h22m21.994697942s) [noise=true , address=81.171.14.70:4027 , piece_id=4FDFCCWF3EIROQAFLOHGS743523OYGSXWQSPQEVD3AMQ6WKDQI7Q, seg_pos={0 0}, upload=256586]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122cYvgTxSvC2aEXnBkRk2o8ARTBXj9gqPoa12vW8hogSN6kpA4): protocol: expected piece hash; context canceled; EOF Nov 21 11:26:23.443237770 (291h22m21.994708231s) [noise=true , address=81.171.14.70:4027 , piece_id=4FDFCCWF3EIROQAFLOHGS743523OYGSXWQSPQEVD3AMQ6WKDQI7Q, seg_pos={0 0}, upload=256586]: Long tail cancellation due to optimal threshold Nov 21 11:26:23.443274720 (291h22m21.994745181s) [seg_pos={0 0}, upload=256586]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Nov 21 11:26:23.443378929 (291h22m21.99484939s) [upload=256586]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 21 11:26:23.443387044 (291h22m21.994857495s) [upload=256586]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 21 11:26:23.685139626 (291h22m22.236610077s) [upload=256586]: Done uploading segments... Nov 21 11:26:23.685160676 (291h22m22.236631127s) [upload=256586]: Upload finished. err: <nil> Duration: 5.893295293s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 2 out of 60 is failed (threshold=3) [.......................X.X..................................] duration: 6.019472626s trace: https://jaeger.dev.storj.io/trace/a7f69ee106efb7f FAIL (2025-11-21 11:48:34 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== upload=256586]: Uploading piece... Nov 21 11:25:27.210346024 (291h21m25.761816485s) [noise=true , address=89.149.201.147:4008 , piece_id=HCX7I4ICPEHDWTYDFIQMKZ5EE5CGPQLYAIHF7WZMTUUU6CNZPHTA, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.211425289 (291h21m25.76289575s) [noise=true , address=81.171.14.70:4044 , piece_id=QYUBNGS2FAFHIWMW7GAGYJ3UVIWVP62LIDCOK6QUXEIQNIXRJDAQ, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.213103795 (291h21m25.764574246s) [noise=true , address=185.167.204.205:4022 , piece_id=BES3YK6PYQV2AKOWSAGHF7DGGWGWSWPIJU2ICAQ5WBHKZ5VT5L6A, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.213408395 (291h21m25.764878856s) [noise=true , address=185.167.204.205:4027 , piece_id=IVPONKBF2PIHZQAW3L4YA4HOMQV5BQML6SKKCJLIEGT5SZSQ2AOA, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.214501747 (291h21m25.765972208s) [noise=true , address=81.171.14.70:4042 , piece_id=XNREZEUZ5L5KZ3L7ZOL6IRK7IHU6DJN4RLEQZRFMDKDH2JWC24CQ, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.215013752 (291h21m25.766484213s) [noise=true , address=81.171.14.70:4034 , piece_id=ITRLKYVNFSYBLABNGNFO3HRJNMX3MWB345FIMWSIPBNAVZPA47UA, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.215087572 (291h21m25.766558044s) [noise=true , address=185.167.204.205:4031 , piece_id=ZEI2YSIQC655R7WS43WADXGFT5RKT5LFSN6IZEQ6SHTDVOV7BZBQ, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.216432844 (291h21m25.767903305s) [noise=true , address=185.167.204.206:4056 , piece_id=EAMZIK6AVJL7IMFF5VOLW3SMCGYKSKB2VXJTNUNLJCVIT7EKXNMQ, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.216662281 (291h21m25.768132742s) [noise=true , address=89.149.201.147:4011 , piece_id=ZATNMD5HOK35AWGJJ5V66VDJ4YGUVZQJ2OBERQL32ZV5GW2HBSMQ, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.218008314 (291h21m25.769478775s) [noise=true , address=81.171.14.70:4032 , piece_id=CFH2RTR3BLBUB5ZIZCFC6UHT3FHT3B2XSCHQMKXNGDJJGPHEMCTQ, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:28.035150976 (291h21m26.586621437s) [upload=256586]: Next returned nil segment Nov 21 11:25:28.035169622 (291h21m26.586640083s) [upload=256586]: Waiting for error group managing segments... Nov 21 11:25:28.416646836 (291h21m26.968117317s) [noise=true , address=185.167.204.206:4064 , piece_id=SRFIWFQLRAGCNIKYLHMGT4QRUL3WH2QPLLIKAJ7DLFJEQUW5DPPA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.417338854 (291h21m26.968809325s) [noise=true , address=185.167.204.205:4033 , piece_id=FEWQVHPIKN3Q6O5AZ6AGJBLUPZ34PBZFB5X3XNSKGPOGI4AIO7JA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.421089406 (291h21m26.972559867s) [noise=true , address=185.167.204.206:4072 , piece_id=GYCO53OWGOMOHK4FHC5GENN4LTQGSZKZKOI5V534RLV3LHLPKC6Q, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.426252887 (291h21m26.977723348s) [noise=true , address=185.167.204.206:4046 , piece_id=BZ57ZA35ITEOA6CAZJGNODK5MUTXEZWDV7E2OGBZOSMS4OQZYAEA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.431720689 (291h21m26.98319115s) [noise=true , address=185.167.204.205:4027 , piece_id=IVPONKBF2PIHZQAW3L4YA4HOMQV5BQML6SKKCJLIEGT5SZSQ2AOA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.438243951 (291h21m26.989714412s) [noise=true , address=185.167.204.205:4022 , piece_id=BES3YK6PYQV2AKOWSAGHF7DGGWGWSWPIJU2ICAQ5WBHKZ5VT5L6A, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.443364331 (291h21m26.994834792s) [noise=true , address=185.167.204.206:4056 , piece_id=EAMZIK6AVJL7IMFF5VOLW3SMCGYKSKB2VXJTNUNLJCVIT7EKXNMQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.445594728 (291h21m26.997065179s) [noise=true , address=185.167.204.205:4016 , piece_id=EHQS3546UXRXTL2MGFCSRTPDDPAYXEYLMISPP5AWY337MHX7HFLQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.449325822 (291h21m27.000796283s) [noise=true , address=185.167.204.205:4031 , piece_id=ZEI2YSIQC655R7WS43WADXGFT5RKT5LFSN6IZEQ6SHTDVOV7BZBQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.449761211 (291h21m27.001231672s) [noise=true , address=185.167.204.205:4018 , piece_id=7OSLPWJLKFTIJPZEAFMZOAH6JAAMX6HKROI25FODFYBPNJWSMLWA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.450419845 (291h21m27.001890306s) [noise=true , address=185.167.204.206:4062 , piece_id=UF3S6NAVGS2KQJPGA5WXP7YG4RQO44P445SNPCWXKXMONZAYUTNQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.453201252 (291h21m27.004671713s) [noise=true , address=185.167.204.205:4007 , piece_id=MEBGOPZ6MFMQY4UZARS2LKJKEMNN5YGSLKPJGTWSKS45CUNXSRGQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:29.661209626 (291h21m28.212680087s) [noise=true , address=89.149.201.147:4016 , piece_id=TGRWGGGBKDANC7YKROWPRMIYNAVFEVE3RXOSUOINFI7KTRS2IYUQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:29.693169182 (291h21m28.244639643s) [noise=true , address=89.149.201.147:4002 , piece_id=UIYJSCMY3LCLRHZUJB6JFJHHXKMYTVKERTZW7F7X44MWJETPY6XA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:29.721204497 (291h21m28.272674958s) [noise=true , address=89.149.201.147:4008 , piece_id=HCX7I4ICPEHDWTYDFIQMKZ5EE5CGPQLYAIHF7WZMTUUU6CNZPHTA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:29.764584711 (291h21m28.316055172s) [noise=true , address=89.149.201.147:4001 , piece_id=QUS6NB3NQBLJFB6YCK3EBDPKCTWEWGKAT3NOR24TVI65SQKHWDOQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:29.768284075 (291h21m28.319754536s) [noise=true , address=89.149.201.147:4011 , piece_id=ZATNMD5HOK35AWGJJ5V66VDJ4YGUVZQJ2OBERQL32ZV5GW2HBSMQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:29.789681961 (291h21m28.341152422s) [noise=true , address=89.149.201.147:4003 , piece_id=RUAYGN2RGF5JJTWFEGIHWH3BHEUDSS5DN2QJ6YLSHPF5BGUXD3RA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:37.298598682 (291h21m35.850069143s) [noise=true , address=81.171.14.70:4043 , piece_id=T3NF6JR3VJIGDMPNUBLMJ3AOTQP5IHOGJYORMCSS7ZMOENQETZ7A, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:37.497478356 (291h21m36.048948818s) [noise=true , address=81.171.14.70:4032 , piece_id=CFH2RTR3BLBUB5ZIZCFC6UHT3FHT3B2XSCHQMKXNGDJJGPHEMCTQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:40.406166933 (291h21m38.957637394s) [noise=true , address=81.171.14.70:4045 , piece_id=BRWQ7G3E547PJE2RLGSESIB3HMP5C4M5NSKAJNGWSFXUBEO6SYEQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:42.511041157 (291h21m41.062511608s) [noise=true , address=81.171.14.70:4029 , piece_id=DKJM5E7EY7LOSBSEBQOZRC6CI5ZPPE47HFOFQJROPCUMUV567CAA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:26:12.012682521 (291h22m10.564152982s) [noise=true , address=81.171.14.70:4042 , piece_id=XNREZEUZ5L5KZ3L7ZOL6IRK7IHU6DJN4RLEQZRFMDKDH2JWC24CQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:26:15.596046283 (291h22m14.147516744s) [noise=true , address=81.171.14.70:4033 , piece_id=JRKPD4EP6FZM5HISOIDJYZY7FAWWQCADAS47FGW2JM5A7RFJR2BQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:26:18.453059659 (291h22m17.00453012s) [noise=true , address=81.171.14.70:4046 , piece_id=OSONRSF6KJS7F2QJVCC3FPZK4AV7QPG5YSM47CW5LIW7EME44AWA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:26:23.442583344 (291h22m21.994053805s) [noise=true , address=81.171.14.70:4034 , piece_id=ITRLKYVNFSYBLABNGNFO3HRJNMX3MWB345FIMWSIPBNAVZPA47UA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:26:23.442621967 (291h22m21.994092428s) [seg_pos={0 0}, upload=256586]: Segment reached optimal threshold of 26 pieces. Nov 21 11:26:23.442836976 (291h22m21.994307437s) [noise=true , address=81.171.14.70:4044 , piece_id=QYUBNGS2FAFHIWMW7GAGYJ3UVIWVP62LIDCOK6QUXEIQNIXRJDAQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1MUVGecysnTSFcJoHUL7QMWb3h6dtXgUd7FbA672fQCyEWhePr): protocol: expected piece hash; context canceled; EOF Nov 21 11:26:23.442845934 (291h22m21.994316395s) [noise=true , address=81.171.14.70:4044 , piece_id=QYUBNGS2FAFHIWMW7GAGYJ3UVIWVP62LIDCOK6QUXEIQNIXRJDAQ, seg_pos={0 0}, upload=256586]: Long tail cancellation due to optimal threshold Nov 21 11:26:23.443227491 (291h22m21.994697942s) [noise=true , address=81.171.14.70:4027 , piece_id=4FDFCCWF3EIROQAFLOHGS743523OYGSXWQSPQEVD3AMQ6WKDQI7Q, seg_pos={0 0}, upload=256586]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122cYvgTxSvC2aEXnBkRk2o8ARTBXj9gqPoa12vW8hogSN6kpA4): protocol: expected piece hash; context canceled; EOF Nov 21 11:26:23.443237770 (291h22m21.994708231s) [noise=true , address=81.171.14.70:4027 , piece_id=4FDFCCWF3EIROQAFLOHGS743523OYGSXWQSPQEVD3AMQ6WKDQI7Q, seg_pos={0 0}, upload=256586]: Long tail cancellation due to optimal threshold Nov 21 11:26:23.443274720 (291h22m21.994745181s) [seg_pos={0 0}, upload=256586]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Nov 21 11:26:23.443378929 (291h22m21.99484939s) [upload=256586]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 21 11:26:23.443387044 (291h22m21.994857495s) [upload=256586]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 21 11:26:23.685139626 (291h22m22.236610077s) [upload=256586]: Done uploading segments... Nov 21 11:26:23.685160676 (291h22m22.236631127s) [upload=256586]: Upload finished. err: <nil> Duration: 6.019505218s core uplink eu-select-1 eu1-placement-15
checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.eu1.linksharing) antiblip: 1 out of 60 is failed (threshold=6) [X...........................................................] duration: 268.22969ms remote: "tcp:207.211.208.129:443" target: https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt trace: https://jaeger.dev.storj.io/trace/e2bae25cf885024 FAIL (2025-11-21 11:48:19 UTC) orbiter warning: Get "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt": context deadline exceeded Duration: 268.27194ms edge linksharing bgp eu1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 46 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX..............] duration: 3.394621163s trace: https://jaeger.dev.storj.io/trace/6af15a3a67b655d5 FAIL (2025-11-21 11:48:16 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 3.394648133s core uplink us-select-1 us1-placement-20
checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us1.gateway) antiblip: 1 out of 60 is failed (threshold=3) [..X.........................................................] duration: 261.255279ms remote: "tcp:207.211.208.129:443" target: https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222 trace: https://jaeger.dev.storj.io/trace/3e5c193367f9a974 FAIL (2025-11-21 11:48:15 UTC) orbiter warning: Get "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222": context deadline exceeded Duration: 261.286339ms edge gateway vhost bgp us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 16 out of 60 is failed (threshold=3) [.X...X.X......X.XX...X.X.X....X............X....X.X.X..X.X..] duration: 6.335998191s trace: https://jaeger.dev.storj.io/trace/65a779f78dabbb29 FAIL (2025-11-21 11:48:09 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 21 11:47:12.819903186 (291h43m11.371373647s) [upload=256902]: Starting upload Nov 21 11:47:12.820012584 (291h43m11.371483046s) [upload=256902]: Done starting upload Nov 21 11:47:12.820071497 (291h43m11.371541968s) [upload=256902]: Uploading segments... Nov 21 11:47:12.820215882 (291h43m11.371686343s) [upload=256902]: Got next segment. Inline: false Nov 21 11:47:12.820319910 (291h43m11.371790371s) [upload=256902]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 21 11:47:12.820331062 (291h43m11.371801513s) [upload=256902]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 21 11:47:13.068398550 (291h43m11.619869011s) [seg_pos={0 0}, upload=256902]: Begin upload segment... Nov 21 11:47:13.068462801 (291h43m11.619933252s) [seg_pos={0 0}, upload=256902]: Done begin upload segment. Nov 21 11:47:13.068483020 (291h43m11.619953481s) [noise=true , address=45.91.116.9:4263 , piece_id=AXRAAACLPJ4WNPJBKUNKUVQKARO7LDKHVM6OAAOBNWIT3XCMMT3Q, seg_pos={0 0}, upload=256902]: Uploading piece... Nov 21 11:47:13.068509531 (291h43m11.619979992s) [noise=true , address=45.91.116.4:4940 , piece_id=7VNMALO44TK6IHSO2RNURGOFWUP7VQ2WMZHDXNSE55SKD2URABFA, seg_pos={0 0}, upload=256902]: Uploading piece... Nov 21 11:47:13.069171702 (291h43m11.620642163s) [noise=true , address=145.79.227.21:4473 , piece_id=FLFHFBWPHSRRDYQ2QOCJW67GNPTCWROGBENFA5B36D2XSWBHRXCA, seg_pos={0 0}, upload=256902]: Uploading piece... Nov 21 11:47:13.069525025 (291h43m11.620995486s) [noise=true , address=77.90.31.13:4626 , piece_id=5C4G5PQMQ4T4ISP3WDKQIWXBHEXOL4HMBF3VJEW67EMPS7ITYV6A, seg_pos={0 0}, upload=256902]: Uploading piece... Nov 21 11:47:13.069954552 (291h43m11.621425013s) [noise=true , address=145.79.227.10:4403 , piece_id=RCWKLIN3X5NZPEDDMST6FMSC4IO7SZPEOL63WZMVUT5ML4ZPMDRQ, seg_pos={0 0}, upload=256902]: Uploading piece... Nov 21 11:47:13.070343192 (291h43m11.621813653s) [noise=true , address=145.79.227.20:4439 , piece_id=EU4ZANKMSD2HADQQZ3KOOOVDQB2WSXVLGJ632KCAYUZB6OJM32KQ, seg_pos={0 0}, upload=256902]: Uploading piece... Nov 21 11:47:13.070743334 (291h43m11.622213795s) [noise=true , address=145.79.227.14:4838 , piece_id=CU46PFV76QK5FETH4AIUNZ4LE7V7UAOEQKDSHWVBIHEAJ3FYMAFA, seg_pos={0 0}, upload=256902]: Uploading piece... Nov 21 11:47:13.071158986 (291h43m11.622629437s) [noise=true , address=145.79.227.26:4654 , piece_id=N5Z4MC2KTCW42HEVFN6ZYFLFL4FZURAR4ZQ6G2UNTYI34LOGILSQ, seg_pos={0 0}, upload=256902]: Uploading piece... Nov 21 11:47:13.071625524 (291h43m11.623095985s) [noise=true , address=45.91.116.10:4081 , piece_id=NIIOOYUQSAYUZM6YEMISHQFHZ7MBUXFMZPKWAED2U2I42CDCHGJQ, seg_pos={0 0}, upload=256902]: Uploading piece... Nov 21 11:47:13.072109045 (291h43m11.623579506s) [noise=true , address=145.79.227.28:4725 , piece_id=FG5TGIUFDMOFS7ZSXMH2FSJOG4CYMXLA44BEBBQ2XYWNQQVDJR7Q, seg_pos={0 0}, upload=256902]: Uploading piece... Nov 21 11:47:13.072642821 (291h43m11.624113272s) [noise=true , address=77.90.31.11:4554 , piece_id=XFDJ6PUFWMANNJW3XW5Y3EJHAJ4IJNAUL6UVICYUPLXYBFUYTEYQ, seg_pos={0 0}, upload=256902]: Uploading piece... Nov 21 11:47:13.074193855 (291h43m11.625664316s) [noise=true , address=145.79.227.8:4330 , piece_id=YA24KHRMYEJS7ZEAMK7DNBKNNSOYNVSETOJ3CKQVHYQ5VFIGV4RA, seg_pos={0 0}, upload=256902]: Uploading piece... Nov 21 11:47:13.075765377 (291h43m11.627235838s) [noise=true , address=45.91.116.5:4299 , piece_id=BVVMC334SJCPAZNZJMRBHNJWWXFJMWFZF7NREWPQBZF7NOA5FNAQ, seg_pos={0 0}, upload=256902]: Uploading piece... Nov 21 11:47:13.077346588 (291h43m11.62881705s) [noise=true , address=77.90.31.19:4842 , piece_id=TI2INXGJJ47N7ERWXHGIEUU5AYXXACSLOHVDY44P7CSYJ6K4XS7A, seg_pos={0 0}, upload=256902]: Uploading piece... Nov 21 11:47:13.079040834 (291h43m11.630511285s) [noise=true , address=45.91.116.8:4048 , piece_id=4FFPIF2BQIFEQ2CMQ25HBPGYXYRTF3KDHMOX4FPN35V3SRPEI4PQ, seg_pos={0 0}, upload=256902]: Uploading piece... Nov 21 11:47:13.080679355 (291h43m11.632149816s) [noise=true , address=145.79.227.9:4367 , piece_id=M3YXW7KNG7KNICBJP7DTIDTGP3IKH6SS7TT2I45FLKIXYCY5T4IA, seg_pos={0 0}, upload=256902]: Uploading piece... Nov 21 11:47:14.687912781 (291h43m13.239383232s) [upload=256902]: Next returned nil segment Nov 21 11:47:14.687929273 (291h43m13.239399724s) [upload=256902]: Waiting for error group managing segments... Nov 21 11:47:15.068852844 (291h43m13.620323295s) [noise=true , address=145.79.227.20:4439 , piece_id=EU4ZANKMSD2HADQQZ3KOOOVDQB2WSXVLGJ632KCAYUZB6OJM32KQ, seg_pos={0 0}, upload=256902]: Done uploading piece. err: <nil> Nov 21 11:47:15.069543749 (291h43m13.62101421s) [noise=true , address=145.79.227.14:4838 , piece_id=CU46PFV76QK5FETH4AIUNZ4LE7V7UAOEQKDSHWVBIHEAJ3FYMAFA, seg_pos={0 0}, upload=256902]: Done uploading piece. err: <nil> Nov 21 11:47:15.089960878 (291h43m13.641431329s) [noise=true , address=145.79.227.26:4654 , piece_id=N5Z4MC2KTCW42HEVFN6ZYFLFL4FZURAR4ZQ6G2UNTYI34LOGILSQ, seg_pos={0 0}, upload=256902]: Done uploading piece. err: <nil> Nov 21 11:47:15.097853837 (291h43m13.649324288s) [noise=true , address=145.79.227.9:4367 , piece_id=M3YXW7KNG7KNICBJP7DTIDTGP3IKH6SS7TT2I45FLKIXYCY5T4IA, seg_pos={0 0}, upload=256902]: Done uploading piece. err: <nil> Nov 21 11:47:15.100087961 (291h43m13.651558412s) [noise=true , address=145.79.227.28:4725 , piece_id=FG5TGIUFDMOFS7ZSXMH2FSJOG4CYMXLA44BEBBQ2XYWNQQVDJR7Q, seg_pos={0 0}, upload=256902]: Done uploading piece. err: <nil> Nov 21 11:47:15.101293046 (291h43m13.652763518s) [noise=true , address=145.79.227.21:4473 , piece_id=FLFHFBWPHSRRDYQ2QOCJW67GNPTCWROGBENFA5B36D2XSWBHRXCA, seg_pos={0 0}, upload=256902]: Done uploading piece. err: <nil> Nov 21 11:47:15.107189524 (291h43m13.658659975s) [noise=true , address=145.79.227.8:4330 , piece_id=YA24KHRMYEJS7ZEAMK7DNBKNNSOYNVSETOJ3CKQVHYQ5VFIGV4RA, seg_pos={0 0}, upload=256902]: Done uploading piece. err: <nil> Nov 21 11:47:15.107226835 (291h43m13.658697296s) [noise=true , address=145.79.227.10:4403 , piece_id=RCWKLIN3X5NZPEDDMST6FMSC4IO7SZPEOL63WZMVUT5ML4ZPMDRQ, seg_pos={0 0}, upload=256902]: Done uploading piece. err: <nil> Nov 21 11:47:15.502447370 (291h43m14.053917831s) [noise=true , address=45.91.116.4:4940 , piece_id=7VNMALO44TK6IHSO2RNURGOFWUP7VQ2WMZHDXNSE55SKD2URABFA, seg_pos={0 0}, upload=256902]: Done uploading piece. err: <nil> Nov 21 11:47:15.506413582 (291h43m14.057884033s) [noise=true , address=45.91.116.9:4263 , piece_id=AXRAAACLPJ4WNPJBKUNKUVQKARO7LDKHVM6OAAOBNWIT3XCMMT3Q, seg_pos={0 0}, upload=256902]: Done uploading piece. err: <nil> Nov 21 11:47:15.534214600 (291h43m14.085685061s) [noise=true , address=77.90.31.13:4626 , piece_id=5C4G5PQMQ4T4ISP3WDKQIWXBHEXOL4HMBF3VJEW67EMPS7ITYV6A, seg_pos={0 0}, upload=256902]: Done uploading piece. err: <nil> Nov 21 11:47:15.546696388 (291h43m14.098166849s) [noise=true , address=45.91.116.5:4299 , piece_id=BVVMC334SJCPAZNZJMRBHNJWWXFJMWFZF7NREWPQBZF7NOA5FNAQ, seg_pos={0 0}, upload=256902]: Done uploading piece. err: <nil> Nov 21 11:47:15.570512259 (291h43m14.12198272s) [noise=true , address=45.91.116.8:4048 , piece_id=4FFPIF2BQIFEQ2CMQ25HBPGYXYRTF3KDHMOX4FPN35V3SRPEI4PQ, seg_pos={0 0}, upload=256902]: Done uploading piece. err: <nil> Nov 21 11:47:15.570536684 (291h43m14.122007146s) [seg_pos={0 0}, upload=256902]: Segment reached optimal threshold of 13 pieces. Nov 21 11:47:15.570778365 (291h43m14.122248826s) [noise=true , address=45.91.116.10:4081 , piece_id=NIIOOYUQSAYUZM6YEMISHQFHZ7MBUXFMZPKWAED2U2I42CDCHGJQ, seg_pos={0 0}, upload=256902]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG): protocol: expected piece hash; context canceled; EOF Nov 21 11:47:15.570790168 (291h43m14.122260639s) [noise=true , address=45.91.116.10:4081 , piece_id=NIIOOYUQSAYUZM6YEMISHQFHZ7MBUXFMZPKWAED2U2I42CDCHGJQ, seg_pos={0 0}, upload=256902]: Long tail cancellation due to optimal threshold Nov 21 11:47:15.570825575 (291h43m14.122296036s) [noise=true , address=77.90.31.11:4554 , piece_id=XFDJ6PUFWMANNJW3XW5Y3EJHAJ4IJNAUL6UVICYUPLXYBFUYTEYQ, seg_pos={0 0}, upload=256902]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek): protocol: expected piece hash; context canceled; EOF Nov 21 11:47:15.570836025 (291h43m14.122306486s) [noise=true , address=77.90.31.11:4554 , piece_id=XFDJ6PUFWMANNJW3XW5Y3EJHAJ4IJNAUL6UVICYUPLXYBFUYTEYQ, seg_pos={0 0}, upload=256902]: Long tail cancellation due to optimal threshold Nov 21 11:47:15.571010367 (291h43m14.122480828s) [noise=true , address=77.90.31.19:4842 , piece_id=TI2INXGJJ47N7ERWXHGIEUU5AYXXACSLOHVDY44P7CSYJ6K4XS7A, seg_pos={0 0}, upload=256902]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3): protocol: write tcp 188.166.225.170:38818->77.90.31.19:4842: use of closed network connection; write tcp 188.166.225.170:38818->77.90.31.19:4842: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38818->77.90.31.19:4842: use of closed network connection Nov 21 11:47:15.571018682 (291h43m14.122489143s) [noise=true , address=77.90.31.19:4842 , piece_id=TI2INXGJJ47N7ERWXHGIEUU5AYXXACSLOHVDY44P7CSYJ6K4XS7A, seg_pos={0 0}, upload=256902]: Long tail cancellation due to optimal threshold Nov 21 11:47:15.571039512 (291h43m14.122509974s) [seg_pos={0 0}, upload=256902]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 21 11:47:15.571117320 (291h43m14.122587771s) [upload=256902]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 21 11:47:15.571122811 (291h43m14.122593262s) [upload=256902]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 21 11:47:15.870951732 (291h43m14.422422183s) [upload=256902]: Done uploading segments... Nov 21 11:47:15.870989133 (291h43m14.422459594s) [upload=256902]: Upload finished. err: <nil> Duration: 6.336025553s core uplink us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 2 out of 60 is failed (threshold=3) [........................X.X.................................] duration: 8.723062147s trace: https://jaeger.dev.storj.io/trace/a7f69ee106efb7f FAIL (2025-11-21 11:48:02 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== upload=256586]: Uploading piece... Nov 21 11:25:27.210346024 (291h21m25.761816485s) [noise=true , address=89.149.201.147:4008 , piece_id=HCX7I4ICPEHDWTYDFIQMKZ5EE5CGPQLYAIHF7WZMTUUU6CNZPHTA, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.211425289 (291h21m25.76289575s) [noise=true , address=81.171.14.70:4044 , piece_id=QYUBNGS2FAFHIWMW7GAGYJ3UVIWVP62LIDCOK6QUXEIQNIXRJDAQ, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.213103795 (291h21m25.764574246s) [noise=true , address=185.167.204.205:4022 , piece_id=BES3YK6PYQV2AKOWSAGHF7DGGWGWSWPIJU2ICAQ5WBHKZ5VT5L6A, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.213408395 (291h21m25.764878856s) [noise=true , address=185.167.204.205:4027 , piece_id=IVPONKBF2PIHZQAW3L4YA4HOMQV5BQML6SKKCJLIEGT5SZSQ2AOA, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.214501747 (291h21m25.765972208s) [noise=true , address=81.171.14.70:4042 , piece_id=XNREZEUZ5L5KZ3L7ZOL6IRK7IHU6DJN4RLEQZRFMDKDH2JWC24CQ, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.215013752 (291h21m25.766484213s) [noise=true , address=81.171.14.70:4034 , piece_id=ITRLKYVNFSYBLABNGNFO3HRJNMX3MWB345FIMWSIPBNAVZPA47UA, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.215087572 (291h21m25.766558044s) [noise=true , address=185.167.204.205:4031 , piece_id=ZEI2YSIQC655R7WS43WADXGFT5RKT5LFSN6IZEQ6SHTDVOV7BZBQ, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.216432844 (291h21m25.767903305s) [noise=true , address=185.167.204.206:4056 , piece_id=EAMZIK6AVJL7IMFF5VOLW3SMCGYKSKB2VXJTNUNLJCVIT7EKXNMQ, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.216662281 (291h21m25.768132742s) [noise=true , address=89.149.201.147:4011 , piece_id=ZATNMD5HOK35AWGJJ5V66VDJ4YGUVZQJ2OBERQL32ZV5GW2HBSMQ, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.218008314 (291h21m25.769478775s) [noise=true , address=81.171.14.70:4032 , piece_id=CFH2RTR3BLBUB5ZIZCFC6UHT3FHT3B2XSCHQMKXNGDJJGPHEMCTQ, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:28.035150976 (291h21m26.586621437s) [upload=256586]: Next returned nil segment Nov 21 11:25:28.035169622 (291h21m26.586640083s) [upload=256586]: Waiting for error group managing segments... Nov 21 11:25:28.416646836 (291h21m26.968117317s) [noise=true , address=185.167.204.206:4064 , piece_id=SRFIWFQLRAGCNIKYLHMGT4QRUL3WH2QPLLIKAJ7DLFJEQUW5DPPA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.417338854 (291h21m26.968809325s) [noise=true , address=185.167.204.205:4033 , piece_id=FEWQVHPIKN3Q6O5AZ6AGJBLUPZ34PBZFB5X3XNSKGPOGI4AIO7JA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.421089406 (291h21m26.972559867s) [noise=true , address=185.167.204.206:4072 , piece_id=GYCO53OWGOMOHK4FHC5GENN4LTQGSZKZKOI5V534RLV3LHLPKC6Q, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.426252887 (291h21m26.977723348s) [noise=true , address=185.167.204.206:4046 , piece_id=BZ57ZA35ITEOA6CAZJGNODK5MUTXEZWDV7E2OGBZOSMS4OQZYAEA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.431720689 (291h21m26.98319115s) [noise=true , address=185.167.204.205:4027 , piece_id=IVPONKBF2PIHZQAW3L4YA4HOMQV5BQML6SKKCJLIEGT5SZSQ2AOA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.438243951 (291h21m26.989714412s) [noise=true , address=185.167.204.205:4022 , piece_id=BES3YK6PYQV2AKOWSAGHF7DGGWGWSWPIJU2ICAQ5WBHKZ5VT5L6A, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.443364331 (291h21m26.994834792s) [noise=true , address=185.167.204.206:4056 , piece_id=EAMZIK6AVJL7IMFF5VOLW3SMCGYKSKB2VXJTNUNLJCVIT7EKXNMQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.445594728 (291h21m26.997065179s) [noise=true , address=185.167.204.205:4016 , piece_id=EHQS3546UXRXTL2MGFCSRTPDDPAYXEYLMISPP5AWY337MHX7HFLQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.449325822 (291h21m27.000796283s) [noise=true , address=185.167.204.205:4031 , piece_id=ZEI2YSIQC655R7WS43WADXGFT5RKT5LFSN6IZEQ6SHTDVOV7BZBQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.449761211 (291h21m27.001231672s) [noise=true , address=185.167.204.205:4018 , piece_id=7OSLPWJLKFTIJPZEAFMZOAH6JAAMX6HKROI25FODFYBPNJWSMLWA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.450419845 (291h21m27.001890306s) [noise=true , address=185.167.204.206:4062 , piece_id=UF3S6NAVGS2KQJPGA5WXP7YG4RQO44P445SNPCWXKXMONZAYUTNQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.453201252 (291h21m27.004671713s) [noise=true , address=185.167.204.205:4007 , piece_id=MEBGOPZ6MFMQY4UZARS2LKJKEMNN5YGSLKPJGTWSKS45CUNXSRGQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:29.661209626 (291h21m28.212680087s) [noise=true , address=89.149.201.147:4016 , piece_id=TGRWGGGBKDANC7YKROWPRMIYNAVFEVE3RXOSUOINFI7KTRS2IYUQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:29.693169182 (291h21m28.244639643s) [noise=true , address=89.149.201.147:4002 , piece_id=UIYJSCMY3LCLRHZUJB6JFJHHXKMYTVKERTZW7F7X44MWJETPY6XA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:29.721204497 (291h21m28.272674958s) [noise=true , address=89.149.201.147:4008 , piece_id=HCX7I4ICPEHDWTYDFIQMKZ5EE5CGPQLYAIHF7WZMTUUU6CNZPHTA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:29.764584711 (291h21m28.316055172s) [noise=true , address=89.149.201.147:4001 , piece_id=QUS6NB3NQBLJFB6YCK3EBDPKCTWEWGKAT3NOR24TVI65SQKHWDOQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:29.768284075 (291h21m28.319754536s) [noise=true , address=89.149.201.147:4011 , piece_id=ZATNMD5HOK35AWGJJ5V66VDJ4YGUVZQJ2OBERQL32ZV5GW2HBSMQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:29.789681961 (291h21m28.341152422s) [noise=true , address=89.149.201.147:4003 , piece_id=RUAYGN2RGF5JJTWFEGIHWH3BHEUDSS5DN2QJ6YLSHPF5BGUXD3RA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:37.298598682 (291h21m35.850069143s) [noise=true , address=81.171.14.70:4043 , piece_id=T3NF6JR3VJIGDMPNUBLMJ3AOTQP5IHOGJYORMCSS7ZMOENQETZ7A, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:37.497478356 (291h21m36.048948818s) [noise=true , address=81.171.14.70:4032 , piece_id=CFH2RTR3BLBUB5ZIZCFC6UHT3FHT3B2XSCHQMKXNGDJJGPHEMCTQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:40.406166933 (291h21m38.957637394s) [noise=true , address=81.171.14.70:4045 , piece_id=BRWQ7G3E547PJE2RLGSESIB3HMP5C4M5NSKAJNGWSFXUBEO6SYEQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:42.511041157 (291h21m41.062511608s) [noise=true , address=81.171.14.70:4029 , piece_id=DKJM5E7EY7LOSBSEBQOZRC6CI5ZPPE47HFOFQJROPCUMUV567CAA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:26:12.012682521 (291h22m10.564152982s) [noise=true , address=81.171.14.70:4042 , piece_id=XNREZEUZ5L5KZ3L7ZOL6IRK7IHU6DJN4RLEQZRFMDKDH2JWC24CQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:26:15.596046283 (291h22m14.147516744s) [noise=true , address=81.171.14.70:4033 , piece_id=JRKPD4EP6FZM5HISOIDJYZY7FAWWQCADAS47FGW2JM5A7RFJR2BQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:26:18.453059659 (291h22m17.00453012s) [noise=true , address=81.171.14.70:4046 , piece_id=OSONRSF6KJS7F2QJVCC3FPZK4AV7QPG5YSM47CW5LIW7EME44AWA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:26:23.442583344 (291h22m21.994053805s) [noise=true , address=81.171.14.70:4034 , piece_id=ITRLKYVNFSYBLABNGNFO3HRJNMX3MWB345FIMWSIPBNAVZPA47UA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:26:23.442621967 (291h22m21.994092428s) [seg_pos={0 0}, upload=256586]: Segment reached optimal threshold of 26 pieces. Nov 21 11:26:23.442836976 (291h22m21.994307437s) [noise=true , address=81.171.14.70:4044 , piece_id=QYUBNGS2FAFHIWMW7GAGYJ3UVIWVP62LIDCOK6QUXEIQNIXRJDAQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1MUVGecysnTSFcJoHUL7QMWb3h6dtXgUd7FbA672fQCyEWhePr): protocol: expected piece hash; context canceled; EOF Nov 21 11:26:23.442845934 (291h22m21.994316395s) [noise=true , address=81.171.14.70:4044 , piece_id=QYUBNGS2FAFHIWMW7GAGYJ3UVIWVP62LIDCOK6QUXEIQNIXRJDAQ, seg_pos={0 0}, upload=256586]: Long tail cancellation due to optimal threshold Nov 21 11:26:23.443227491 (291h22m21.994697942s) [noise=true , address=81.171.14.70:4027 , piece_id=4FDFCCWF3EIROQAFLOHGS743523OYGSXWQSPQEVD3AMQ6WKDQI7Q, seg_pos={0 0}, upload=256586]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122cYvgTxSvC2aEXnBkRk2o8ARTBXj9gqPoa12vW8hogSN6kpA4): protocol: expected piece hash; context canceled; EOF Nov 21 11:26:23.443237770 (291h22m21.994708231s) [noise=true , address=81.171.14.70:4027 , piece_id=4FDFCCWF3EIROQAFLOHGS743523OYGSXWQSPQEVD3AMQ6WKDQI7Q, seg_pos={0 0}, upload=256586]: Long tail cancellation due to optimal threshold Nov 21 11:26:23.443274720 (291h22m21.994745181s) [seg_pos={0 0}, upload=256586]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Nov 21 11:26:23.443378929 (291h22m21.99484939s) [upload=256586]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 21 11:26:23.443387044 (291h22m21.994857495s) [upload=256586]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 21 11:26:23.685139626 (291h22m22.236610077s) [upload=256586]: Done uploading segments... Nov 21 11:26:23.685160676 (291h22m22.236631127s) [upload=256586]: Upload finished. err: <nil> Duration: 8.723088846s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 16 out of 60 is failed (threshold=3) [..X...X.X......X.XX...X.X.X....X............X....X.X.X..X.X.] duration: 13.098538703s trace: https://jaeger.dev.storj.io/trace/65a779f78dabbb29 FAIL (2025-11-21 11:47:55 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 21 11:47:12.819903186 (291h43m11.371373647s) [upload=256902]: Starting upload Nov 21 11:47:12.820012584 (291h43m11.371483046s) [upload=256902]: Done starting upload Nov 21 11:47:12.820071497 (291h43m11.371541968s) [upload=256902]: Uploading segments... Nov 21 11:47:12.820215882 (291h43m11.371686343s) [upload=256902]: Got next segment. Inline: false Nov 21 11:47:12.820319910 (291h43m11.371790371s) [upload=256902]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 21 11:47:12.820331062 (291h43m11.371801513s) [upload=256902]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 21 11:47:13.068398550 (291h43m11.619869011s) [seg_pos={0 0}, upload=256902]: Begin upload segment... Nov 21 11:47:13.068462801 (291h43m11.619933252s) [seg_pos={0 0}, upload=256902]: Done begin upload segment. Nov 21 11:47:13.068483020 (291h43m11.619953481s) [noise=true , address=45.91.116.9:4263 , piece_id=AXRAAACLPJ4WNPJBKUNKUVQKARO7LDKHVM6OAAOBNWIT3XCMMT3Q, seg_pos={0 0}, upload=256902]: Uploading piece... Nov 21 11:47:13.068509531 (291h43m11.619979992s) [noise=true , address=45.91.116.4:4940 , piece_id=7VNMALO44TK6IHSO2RNURGOFWUP7VQ2WMZHDXNSE55SKD2URABFA, seg_pos={0 0}, upload=256902]: Uploading piece... Nov 21 11:47:13.069171702 (291h43m11.620642163s) [noise=true , address=145.79.227.21:4473 , piece_id=FLFHFBWPHSRRDYQ2QOCJW67GNPTCWROGBENFA5B36D2XSWBHRXCA, seg_pos={0 0}, upload=256902]: Uploading piece... Nov 21 11:47:13.069525025 (291h43m11.620995486s) [noise=true , address=77.90.31.13:4626 , piece_id=5C4G5PQMQ4T4ISP3WDKQIWXBHEXOL4HMBF3VJEW67EMPS7ITYV6A, seg_pos={0 0}, upload=256902]: Uploading piece... Nov 21 11:47:13.069954552 (291h43m11.621425013s) [noise=true , address=145.79.227.10:4403 , piece_id=RCWKLIN3X5NZPEDDMST6FMSC4IO7SZPEOL63WZMVUT5ML4ZPMDRQ, seg_pos={0 0}, upload=256902]: Uploading piece... Nov 21 11:47:13.070343192 (291h43m11.621813653s) [noise=true , address=145.79.227.20:4439 , piece_id=EU4ZANKMSD2HADQQZ3KOOOVDQB2WSXVLGJ632KCAYUZB6OJM32KQ, seg_pos={0 0}, upload=256902]: Uploading piece... Nov 21 11:47:13.070743334 (291h43m11.622213795s) [noise=true , address=145.79.227.14:4838 , piece_id=CU46PFV76QK5FETH4AIUNZ4LE7V7UAOEQKDSHWVBIHEAJ3FYMAFA, seg_pos={0 0}, upload=256902]: Uploading piece... Nov 21 11:47:13.071158986 (291h43m11.622629437s) [noise=true , address=145.79.227.26:4654 , piece_id=N5Z4MC2KTCW42HEVFN6ZYFLFL4FZURAR4ZQ6G2UNTYI34LOGILSQ, seg_pos={0 0}, upload=256902]: Uploading piece... Nov 21 11:47:13.071625524 (291h43m11.623095985s) [noise=true , address=45.91.116.10:4081 , piece_id=NIIOOYUQSAYUZM6YEMISHQFHZ7MBUXFMZPKWAED2U2I42CDCHGJQ, seg_pos={0 0}, upload=256902]: Uploading piece... Nov 21 11:47:13.072109045 (291h43m11.623579506s) [noise=true , address=145.79.227.28:4725 , piece_id=FG5TGIUFDMOFS7ZSXMH2FSJOG4CYMXLA44BEBBQ2XYWNQQVDJR7Q, seg_pos={0 0}, upload=256902]: Uploading piece... Nov 21 11:47:13.072642821 (291h43m11.624113272s) [noise=true , address=77.90.31.11:4554 , piece_id=XFDJ6PUFWMANNJW3XW5Y3EJHAJ4IJNAUL6UVICYUPLXYBFUYTEYQ, seg_pos={0 0}, upload=256902]: Uploading piece... Nov 21 11:47:13.074193855 (291h43m11.625664316s) [noise=true , address=145.79.227.8:4330 , piece_id=YA24KHRMYEJS7ZEAMK7DNBKNNSOYNVSETOJ3CKQVHYQ5VFIGV4RA, seg_pos={0 0}, upload=256902]: Uploading piece... Nov 21 11:47:13.075765377 (291h43m11.627235838s) [noise=true , address=45.91.116.5:4299 , piece_id=BVVMC334SJCPAZNZJMRBHNJWWXFJMWFZF7NREWPQBZF7NOA5FNAQ, seg_pos={0 0}, upload=256902]: Uploading piece... Nov 21 11:47:13.077346588 (291h43m11.62881705s) [noise=true , address=77.90.31.19:4842 , piece_id=TI2INXGJJ47N7ERWXHGIEUU5AYXXACSLOHVDY44P7CSYJ6K4XS7A, seg_pos={0 0}, upload=256902]: Uploading piece... Nov 21 11:47:13.079040834 (291h43m11.630511285s) [noise=true , address=45.91.116.8:4048 , piece_id=4FFPIF2BQIFEQ2CMQ25HBPGYXYRTF3KDHMOX4FPN35V3SRPEI4PQ, seg_pos={0 0}, upload=256902]: Uploading piece... Nov 21 11:47:13.080679355 (291h43m11.632149816s) [noise=true , address=145.79.227.9:4367 , piece_id=M3YXW7KNG7KNICBJP7DTIDTGP3IKH6SS7TT2I45FLKIXYCY5T4IA, seg_pos={0 0}, upload=256902]: Uploading piece... Nov 21 11:47:14.687912781 (291h43m13.239383232s) [upload=256902]: Next returned nil segment Nov 21 11:47:14.687929273 (291h43m13.239399724s) [upload=256902]: Waiting for error group managing segments... Nov 21 11:47:15.068852844 (291h43m13.620323295s) [noise=true , address=145.79.227.20:4439 , piece_id=EU4ZANKMSD2HADQQZ3KOOOVDQB2WSXVLGJ632KCAYUZB6OJM32KQ, seg_pos={0 0}, upload=256902]: Done uploading piece. err: <nil> Nov 21 11:47:15.069543749 (291h43m13.62101421s) [noise=true , address=145.79.227.14:4838 , piece_id=CU46PFV76QK5FETH4AIUNZ4LE7V7UAOEQKDSHWVBIHEAJ3FYMAFA, seg_pos={0 0}, upload=256902]: Done uploading piece. err: <nil> Nov 21 11:47:15.089960878 (291h43m13.641431329s) [noise=true , address=145.79.227.26:4654 , piece_id=N5Z4MC2KTCW42HEVFN6ZYFLFL4FZURAR4ZQ6G2UNTYI34LOGILSQ, seg_pos={0 0}, upload=256902]: Done uploading piece. err: <nil> Nov 21 11:47:15.097853837 (291h43m13.649324288s) [noise=true , address=145.79.227.9:4367 , piece_id=M3YXW7KNG7KNICBJP7DTIDTGP3IKH6SS7TT2I45FLKIXYCY5T4IA, seg_pos={0 0}, upload=256902]: Done uploading piece. err: <nil> Nov 21 11:47:15.100087961 (291h43m13.651558412s) [noise=true , address=145.79.227.28:4725 , piece_id=FG5TGIUFDMOFS7ZSXMH2FSJOG4CYMXLA44BEBBQ2XYWNQQVDJR7Q, seg_pos={0 0}, upload=256902]: Done uploading piece. err: <nil> Nov 21 11:47:15.101293046 (291h43m13.652763518s) [noise=true , address=145.79.227.21:4473 , piece_id=FLFHFBWPHSRRDYQ2QOCJW67GNPTCWROGBENFA5B36D2XSWBHRXCA, seg_pos={0 0}, upload=256902]: Done uploading piece. err: <nil> Nov 21 11:47:15.107189524 (291h43m13.658659975s) [noise=true , address=145.79.227.8:4330 , piece_id=YA24KHRMYEJS7ZEAMK7DNBKNNSOYNVSETOJ3CKQVHYQ5VFIGV4RA, seg_pos={0 0}, upload=256902]: Done uploading piece. err: <nil> Nov 21 11:47:15.107226835 (291h43m13.658697296s) [noise=true , address=145.79.227.10:4403 , piece_id=RCWKLIN3X5NZPEDDMST6FMSC4IO7SZPEOL63WZMVUT5ML4ZPMDRQ, seg_pos={0 0}, upload=256902]: Done uploading piece. err: <nil> Nov 21 11:47:15.502447370 (291h43m14.053917831s) [noise=true , address=45.91.116.4:4940 , piece_id=7VNMALO44TK6IHSO2RNURGOFWUP7VQ2WMZHDXNSE55SKD2URABFA, seg_pos={0 0}, upload=256902]: Done uploading piece. err: <nil> Nov 21 11:47:15.506413582 (291h43m14.057884033s) [noise=true , address=45.91.116.9:4263 , piece_id=AXRAAACLPJ4WNPJBKUNKUVQKARO7LDKHVM6OAAOBNWIT3XCMMT3Q, seg_pos={0 0}, upload=256902]: Done uploading piece. err: <nil> Nov 21 11:47:15.534214600 (291h43m14.085685061s) [noise=true , address=77.90.31.13:4626 , piece_id=5C4G5PQMQ4T4ISP3WDKQIWXBHEXOL4HMBF3VJEW67EMPS7ITYV6A, seg_pos={0 0}, upload=256902]: Done uploading piece. err: <nil> Nov 21 11:47:15.546696388 (291h43m14.098166849s) [noise=true , address=45.91.116.5:4299 , piece_id=BVVMC334SJCPAZNZJMRBHNJWWXFJMWFZF7NREWPQBZF7NOA5FNAQ, seg_pos={0 0}, upload=256902]: Done uploading piece. err: <nil> Nov 21 11:47:15.570512259 (291h43m14.12198272s) [noise=true , address=45.91.116.8:4048 , piece_id=4FFPIF2BQIFEQ2CMQ25HBPGYXYRTF3KDHMOX4FPN35V3SRPEI4PQ, seg_pos={0 0}, upload=256902]: Done uploading piece. err: <nil> Nov 21 11:47:15.570536684 (291h43m14.122007146s) [seg_pos={0 0}, upload=256902]: Segment reached optimal threshold of 13 pieces. Nov 21 11:47:15.570778365 (291h43m14.122248826s) [noise=true , address=45.91.116.10:4081 , piece_id=NIIOOYUQSAYUZM6YEMISHQFHZ7MBUXFMZPKWAED2U2I42CDCHGJQ, seg_pos={0 0}, upload=256902]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG): protocol: expected piece hash; context canceled; EOF Nov 21 11:47:15.570790168 (291h43m14.122260639s) [noise=true , address=45.91.116.10:4081 , piece_id=NIIOOYUQSAYUZM6YEMISHQFHZ7MBUXFMZPKWAED2U2I42CDCHGJQ, seg_pos={0 0}, upload=256902]: Long tail cancellation due to optimal threshold Nov 21 11:47:15.570825575 (291h43m14.122296036s) [noise=true , address=77.90.31.11:4554 , piece_id=XFDJ6PUFWMANNJW3XW5Y3EJHAJ4IJNAUL6UVICYUPLXYBFUYTEYQ, seg_pos={0 0}, upload=256902]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek): protocol: expected piece hash; context canceled; EOF Nov 21 11:47:15.570836025 (291h43m14.122306486s) [noise=true , address=77.90.31.11:4554 , piece_id=XFDJ6PUFWMANNJW3XW5Y3EJHAJ4IJNAUL6UVICYUPLXYBFUYTEYQ, seg_pos={0 0}, upload=256902]: Long tail cancellation due to optimal threshold Nov 21 11:47:15.571010367 (291h43m14.122480828s) [noise=true , address=77.90.31.19:4842 , piece_id=TI2INXGJJ47N7ERWXHGIEUU5AYXXACSLOHVDY44P7CSYJ6K4XS7A, seg_pos={0 0}, upload=256902]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3): protocol: write tcp 188.166.225.170:38818->77.90.31.19:4842: use of closed network connection; write tcp 188.166.225.170:38818->77.90.31.19:4842: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38818->77.90.31.19:4842: use of closed network connection Nov 21 11:47:15.571018682 (291h43m14.122489143s) [noise=true , address=77.90.31.19:4842 , piece_id=TI2INXGJJ47N7ERWXHGIEUU5AYXXACSLOHVDY44P7CSYJ6K4XS7A, seg_pos={0 0}, upload=256902]: Long tail cancellation due to optimal threshold Nov 21 11:47:15.571039512 (291h43m14.122509974s) [seg_pos={0 0}, upload=256902]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 21 11:47:15.571117320 (291h43m14.122587771s) [upload=256902]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 21 11:47:15.571122811 (291h43m14.122593262s) [upload=256902]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 21 11:47:15.870951732 (291h43m14.422422183s) [upload=256902]: Done uploading segments... Nov 21 11:47:15.870989133 (291h43m14.422459594s) [upload=256902]: Upload finished. err: <nil> Duration: 13.098553641s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 16 out of 60 is failed (threshold=3) [...X...X.X......X.XX...X.X.X....X............X....X.X.X..X.X] duration: 30.002898467s trace: https://jaeger.dev.storj.io/trace/65a779f78dabbb29 FAIL (2025-11-21 11:47:42 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 21 11:47:12.819903186 (291h43m11.371373647s) [upload=256902]: Starting upload Nov 21 11:47:12.820012584 (291h43m11.371483046s) [upload=256902]: Done starting upload Nov 21 11:47:12.820071497 (291h43m11.371541968s) [upload=256902]: Uploading segments... Nov 21 11:47:12.820215882 (291h43m11.371686343s) [upload=256902]: Got next segment. Inline: false Nov 21 11:47:12.820319910 (291h43m11.371790371s) [upload=256902]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 21 11:47:12.820331062 (291h43m11.371801513s) [upload=256902]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 21 11:47:13.068398550 (291h43m11.619869011s) [seg_pos={0 0}, upload=256902]: Begin upload segment... Nov 21 11:47:13.068462801 (291h43m11.619933252s) [seg_pos={0 0}, upload=256902]: Done begin upload segment. Nov 21 11:47:13.068483020 (291h43m11.619953481s) [noise=true , address=45.91.116.9:4263 , piece_id=AXRAAACLPJ4WNPJBKUNKUVQKARO7LDKHVM6OAAOBNWIT3XCMMT3Q, seg_pos={0 0}, upload=256902]: Uploading piece... Nov 21 11:47:13.068509531 (291h43m11.619979992s) [noise=true , address=45.91.116.4:4940 , piece_id=7VNMALO44TK6IHSO2RNURGOFWUP7VQ2WMZHDXNSE55SKD2URABFA, seg_pos={0 0}, upload=256902]: Uploading piece... Nov 21 11:47:13.069171702 (291h43m11.620642163s) [noise=true , address=145.79.227.21:4473 , piece_id=FLFHFBWPHSRRDYQ2QOCJW67GNPTCWROGBENFA5B36D2XSWBHRXCA, seg_pos={0 0}, upload=256902]: Uploading piece... Nov 21 11:47:13.069525025 (291h43m11.620995486s) [noise=true , address=77.90.31.13:4626 , piece_id=5C4G5PQMQ4T4ISP3WDKQIWXBHEXOL4HMBF3VJEW67EMPS7ITYV6A, seg_pos={0 0}, upload=256902]: Uploading piece... Nov 21 11:47:13.069954552 (291h43m11.621425013s) [noise=true , address=145.79.227.10:4403 , piece_id=RCWKLIN3X5NZPEDDMST6FMSC4IO7SZPEOL63WZMVUT5ML4ZPMDRQ, seg_pos={0 0}, upload=256902]: Uploading piece... Nov 21 11:47:13.070343192 (291h43m11.621813653s) [noise=true , address=145.79.227.20:4439 , piece_id=EU4ZANKMSD2HADQQZ3KOOOVDQB2WSXVLGJ632KCAYUZB6OJM32KQ, seg_pos={0 0}, upload=256902]: Uploading piece... Nov 21 11:47:13.070743334 (291h43m11.622213795s) [noise=true , address=145.79.227.14:4838 , piece_id=CU46PFV76QK5FETH4AIUNZ4LE7V7UAOEQKDSHWVBIHEAJ3FYMAFA, seg_pos={0 0}, upload=256902]: Uploading piece... Nov 21 11:47:13.071158986 (291h43m11.622629437s) [noise=true , address=145.79.227.26:4654 , piece_id=N5Z4MC2KTCW42HEVFN6ZYFLFL4FZURAR4ZQ6G2UNTYI34LOGILSQ, seg_pos={0 0}, upload=256902]: Uploading piece... Nov 21 11:47:13.071625524 (291h43m11.623095985s) [noise=true , address=45.91.116.10:4081 , piece_id=NIIOOYUQSAYUZM6YEMISHQFHZ7MBUXFMZPKWAED2U2I42CDCHGJQ, seg_pos={0 0}, upload=256902]: Uploading piece... Nov 21 11:47:13.072109045 (291h43m11.623579506s) [noise=true , address=145.79.227.28:4725 , piece_id=FG5TGIUFDMOFS7ZSXMH2FSJOG4CYMXLA44BEBBQ2XYWNQQVDJR7Q, seg_pos={0 0}, upload=256902]: Uploading piece... Nov 21 11:47:13.072642821 (291h43m11.624113272s) [noise=true , address=77.90.31.11:4554 , piece_id=XFDJ6PUFWMANNJW3XW5Y3EJHAJ4IJNAUL6UVICYUPLXYBFUYTEYQ, seg_pos={0 0}, upload=256902]: Uploading piece... Nov 21 11:47:13.074193855 (291h43m11.625664316s) [noise=true , address=145.79.227.8:4330 , piece_id=YA24KHRMYEJS7ZEAMK7DNBKNNSOYNVSETOJ3CKQVHYQ5VFIGV4RA, seg_pos={0 0}, upload=256902]: Uploading piece... Nov 21 11:47:13.075765377 (291h43m11.627235838s) [noise=true , address=45.91.116.5:4299 , piece_id=BVVMC334SJCPAZNZJMRBHNJWWXFJMWFZF7NREWPQBZF7NOA5FNAQ, seg_pos={0 0}, upload=256902]: Uploading piece... Nov 21 11:47:13.077346588 (291h43m11.62881705s) [noise=true , address=77.90.31.19:4842 , piece_id=TI2INXGJJ47N7ERWXHGIEUU5AYXXACSLOHVDY44P7CSYJ6K4XS7A, seg_pos={0 0}, upload=256902]: Uploading piece... Nov 21 11:47:13.079040834 (291h43m11.630511285s) [noise=true , address=45.91.116.8:4048 , piece_id=4FFPIF2BQIFEQ2CMQ25HBPGYXYRTF3KDHMOX4FPN35V3SRPEI4PQ, seg_pos={0 0}, upload=256902]: Uploading piece... Nov 21 11:47:13.080679355 (291h43m11.632149816s) [noise=true , address=145.79.227.9:4367 , piece_id=M3YXW7KNG7KNICBJP7DTIDTGP3IKH6SS7TT2I45FLKIXYCY5T4IA, seg_pos={0 0}, upload=256902]: Uploading piece... Nov 21 11:47:14.687912781 (291h43m13.239383232s) [upload=256902]: Next returned nil segment Nov 21 11:47:14.687929273 (291h43m13.239399724s) [upload=256902]: Waiting for error group managing segments... Nov 21 11:47:15.068852844 (291h43m13.620323295s) [noise=true , address=145.79.227.20:4439 , piece_id=EU4ZANKMSD2HADQQZ3KOOOVDQB2WSXVLGJ632KCAYUZB6OJM32KQ, seg_pos={0 0}, upload=256902]: Done uploading piece. err: <nil> Nov 21 11:47:15.069543749 (291h43m13.62101421s) [noise=true , address=145.79.227.14:4838 , piece_id=CU46PFV76QK5FETH4AIUNZ4LE7V7UAOEQKDSHWVBIHEAJ3FYMAFA, seg_pos={0 0}, upload=256902]: Done uploading piece. err: <nil> Nov 21 11:47:15.089960878 (291h43m13.641431329s) [noise=true , address=145.79.227.26:4654 , piece_id=N5Z4MC2KTCW42HEVFN6ZYFLFL4FZURAR4ZQ6G2UNTYI34LOGILSQ, seg_pos={0 0}, upload=256902]: Done uploading piece. err: <nil> Nov 21 11:47:15.097853837 (291h43m13.649324288s) [noise=true , address=145.79.227.9:4367 , piece_id=M3YXW7KNG7KNICBJP7DTIDTGP3IKH6SS7TT2I45FLKIXYCY5T4IA, seg_pos={0 0}, upload=256902]: Done uploading piece. err: <nil> Nov 21 11:47:15.100087961 (291h43m13.651558412s) [noise=true , address=145.79.227.28:4725 , piece_id=FG5TGIUFDMOFS7ZSXMH2FSJOG4CYMXLA44BEBBQ2XYWNQQVDJR7Q, seg_pos={0 0}, upload=256902]: Done uploading piece. err: <nil> Nov 21 11:47:15.101293046 (291h43m13.652763518s) [noise=true , address=145.79.227.21:4473 , piece_id=FLFHFBWPHSRRDYQ2QOCJW67GNPTCWROGBENFA5B36D2XSWBHRXCA, seg_pos={0 0}, upload=256902]: Done uploading piece. err: <nil> Nov 21 11:47:15.107189524 (291h43m13.658659975s) [noise=true , address=145.79.227.8:4330 , piece_id=YA24KHRMYEJS7ZEAMK7DNBKNNSOYNVSETOJ3CKQVHYQ5VFIGV4RA, seg_pos={0 0}, upload=256902]: Done uploading piece. err: <nil> Nov 21 11:47:15.107226835 (291h43m13.658697296s) [noise=true , address=145.79.227.10:4403 , piece_id=RCWKLIN3X5NZPEDDMST6FMSC4IO7SZPEOL63WZMVUT5ML4ZPMDRQ, seg_pos={0 0}, upload=256902]: Done uploading piece. err: <nil> Nov 21 11:47:15.502447370 (291h43m14.053917831s) [noise=true , address=45.91.116.4:4940 , piece_id=7VNMALO44TK6IHSO2RNURGOFWUP7VQ2WMZHDXNSE55SKD2URABFA, seg_pos={0 0}, upload=256902]: Done uploading piece. err: <nil> Nov 21 11:47:15.506413582 (291h43m14.057884033s) [noise=true , address=45.91.116.9:4263 , piece_id=AXRAAACLPJ4WNPJBKUNKUVQKARO7LDKHVM6OAAOBNWIT3XCMMT3Q, seg_pos={0 0}, upload=256902]: Done uploading piece. err: <nil> Nov 21 11:47:15.534214600 (291h43m14.085685061s) [noise=true , address=77.90.31.13:4626 , piece_id=5C4G5PQMQ4T4ISP3WDKQIWXBHEXOL4HMBF3VJEW67EMPS7ITYV6A, seg_pos={0 0}, upload=256902]: Done uploading piece. err: <nil> Nov 21 11:47:15.546696388 (291h43m14.098166849s) [noise=true , address=45.91.116.5:4299 , piece_id=BVVMC334SJCPAZNZJMRBHNJWWXFJMWFZF7NREWPQBZF7NOA5FNAQ, seg_pos={0 0}, upload=256902]: Done uploading piece. err: <nil> Nov 21 11:47:15.570512259 (291h43m14.12198272s) [noise=true , address=45.91.116.8:4048 , piece_id=4FFPIF2BQIFEQ2CMQ25HBPGYXYRTF3KDHMOX4FPN35V3SRPEI4PQ, seg_pos={0 0}, upload=256902]: Done uploading piece. err: <nil> Nov 21 11:47:15.570536684 (291h43m14.122007146s) [seg_pos={0 0}, upload=256902]: Segment reached optimal threshold of 13 pieces. Nov 21 11:47:15.570778365 (291h43m14.122248826s) [noise=true , address=45.91.116.10:4081 , piece_id=NIIOOYUQSAYUZM6YEMISHQFHZ7MBUXFMZPKWAED2U2I42CDCHGJQ, seg_pos={0 0}, upload=256902]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG): protocol: expected piece hash; context canceled; EOF Nov 21 11:47:15.570790168 (291h43m14.122260639s) [noise=true , address=45.91.116.10:4081 , piece_id=NIIOOYUQSAYUZM6YEMISHQFHZ7MBUXFMZPKWAED2U2I42CDCHGJQ, seg_pos={0 0}, upload=256902]: Long tail cancellation due to optimal threshold Nov 21 11:47:15.570825575 (291h43m14.122296036s) [noise=true , address=77.90.31.11:4554 , piece_id=XFDJ6PUFWMANNJW3XW5Y3EJHAJ4IJNAUL6UVICYUPLXYBFUYTEYQ, seg_pos={0 0}, upload=256902]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek): protocol: expected piece hash; context canceled; EOF Nov 21 11:47:15.570836025 (291h43m14.122306486s) [noise=true , address=77.90.31.11:4554 , piece_id=XFDJ6PUFWMANNJW3XW5Y3EJHAJ4IJNAUL6UVICYUPLXYBFUYTEYQ, seg_pos={0 0}, upload=256902]: Long tail cancellation due to optimal threshold Nov 21 11:47:15.571010367 (291h43m14.122480828s) [noise=true , address=77.90.31.19:4842 , piece_id=TI2INXGJJ47N7ERWXHGIEUU5AYXXACSLOHVDY44P7CSYJ6K4XS7A, seg_pos={0 0}, upload=256902]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3): protocol: write tcp 188.166.225.170:38818->77.90.31.19:4842: use of closed network connection; write tcp 188.166.225.170:38818->77.90.31.19:4842: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38818->77.90.31.19:4842: use of closed network connection Nov 21 11:47:15.571018682 (291h43m14.122489143s) [noise=true , address=77.90.31.19:4842 , piece_id=TI2INXGJJ47N7ERWXHGIEUU5AYXXACSLOHVDY44P7CSYJ6K4XS7A, seg_pos={0 0}, upload=256902]: Long tail cancellation due to optimal threshold Nov 21 11:47:15.571039512 (291h43m14.122509974s) [seg_pos={0 0}, upload=256902]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 21 11:47:15.571117320 (291h43m14.122587771s) [upload=256902]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 21 11:47:15.571122811 (291h43m14.122593262s) [upload=256902]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 21 11:47:15.870951732 (291h43m14.422422183s) [upload=256902]: Done uploading segments... Nov 21 11:47:15.870989133 (291h43m14.422459594s) [upload=256902]: Upload finished. err: <nil> Duration: 30.002923916s core uplink us-select-1
checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.eu1.linksharing) antiblip: 1 out of 60 is failed (threshold=6) [.X..........................................................] duration: 252.866667ms remote: "tcp:207.211.208.129:443" target: https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt trace: https://jaeger.dev.storj.io/trace/e2bae25cf885024 FAIL (2025-11-21 11:47:35 UTC) orbiter warning: Get "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt": context deadline exceeded Duration: 252.888909ms edge linksharing bgp eu1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 2 out of 60 is failed (threshold=3) [.........................X.X................................] duration: 6.492713549s trace: https://jaeger.dev.storj.io/trace/a7f69ee106efb7f FAIL (2025-11-21 11:47:11 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== upload=256586]: Uploading piece... Nov 21 11:25:27.210346024 (291h21m25.761816485s) [noise=true , address=89.149.201.147:4008 , piece_id=HCX7I4ICPEHDWTYDFIQMKZ5EE5CGPQLYAIHF7WZMTUUU6CNZPHTA, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.211425289 (291h21m25.76289575s) [noise=true , address=81.171.14.70:4044 , piece_id=QYUBNGS2FAFHIWMW7GAGYJ3UVIWVP62LIDCOK6QUXEIQNIXRJDAQ, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.213103795 (291h21m25.764574246s) [noise=true , address=185.167.204.205:4022 , piece_id=BES3YK6PYQV2AKOWSAGHF7DGGWGWSWPIJU2ICAQ5WBHKZ5VT5L6A, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.213408395 (291h21m25.764878856s) [noise=true , address=185.167.204.205:4027 , piece_id=IVPONKBF2PIHZQAW3L4YA4HOMQV5BQML6SKKCJLIEGT5SZSQ2AOA, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.214501747 (291h21m25.765972208s) [noise=true , address=81.171.14.70:4042 , piece_id=XNREZEUZ5L5KZ3L7ZOL6IRK7IHU6DJN4RLEQZRFMDKDH2JWC24CQ, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.215013752 (291h21m25.766484213s) [noise=true , address=81.171.14.70:4034 , piece_id=ITRLKYVNFSYBLABNGNFO3HRJNMX3MWB345FIMWSIPBNAVZPA47UA, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.215087572 (291h21m25.766558044s) [noise=true , address=185.167.204.205:4031 , piece_id=ZEI2YSIQC655R7WS43WADXGFT5RKT5LFSN6IZEQ6SHTDVOV7BZBQ, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.216432844 (291h21m25.767903305s) [noise=true , address=185.167.204.206:4056 , piece_id=EAMZIK6AVJL7IMFF5VOLW3SMCGYKSKB2VXJTNUNLJCVIT7EKXNMQ, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.216662281 (291h21m25.768132742s) [noise=true , address=89.149.201.147:4011 , piece_id=ZATNMD5HOK35AWGJJ5V66VDJ4YGUVZQJ2OBERQL32ZV5GW2HBSMQ, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.218008314 (291h21m25.769478775s) [noise=true , address=81.171.14.70:4032 , piece_id=CFH2RTR3BLBUB5ZIZCFC6UHT3FHT3B2XSCHQMKXNGDJJGPHEMCTQ, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:28.035150976 (291h21m26.586621437s) [upload=256586]: Next returned nil segment Nov 21 11:25:28.035169622 (291h21m26.586640083s) [upload=256586]: Waiting for error group managing segments... Nov 21 11:25:28.416646836 (291h21m26.968117317s) [noise=true , address=185.167.204.206:4064 , piece_id=SRFIWFQLRAGCNIKYLHMGT4QRUL3WH2QPLLIKAJ7DLFJEQUW5DPPA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.417338854 (291h21m26.968809325s) [noise=true , address=185.167.204.205:4033 , piece_id=FEWQVHPIKN3Q6O5AZ6AGJBLUPZ34PBZFB5X3XNSKGPOGI4AIO7JA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.421089406 (291h21m26.972559867s) [noise=true , address=185.167.204.206:4072 , piece_id=GYCO53OWGOMOHK4FHC5GENN4LTQGSZKZKOI5V534RLV3LHLPKC6Q, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.426252887 (291h21m26.977723348s) [noise=true , address=185.167.204.206:4046 , piece_id=BZ57ZA35ITEOA6CAZJGNODK5MUTXEZWDV7E2OGBZOSMS4OQZYAEA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.431720689 (291h21m26.98319115s) [noise=true , address=185.167.204.205:4027 , piece_id=IVPONKBF2PIHZQAW3L4YA4HOMQV5BQML6SKKCJLIEGT5SZSQ2AOA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.438243951 (291h21m26.989714412s) [noise=true , address=185.167.204.205:4022 , piece_id=BES3YK6PYQV2AKOWSAGHF7DGGWGWSWPIJU2ICAQ5WBHKZ5VT5L6A, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.443364331 (291h21m26.994834792s) [noise=true , address=185.167.204.206:4056 , piece_id=EAMZIK6AVJL7IMFF5VOLW3SMCGYKSKB2VXJTNUNLJCVIT7EKXNMQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.445594728 (291h21m26.997065179s) [noise=true , address=185.167.204.205:4016 , piece_id=EHQS3546UXRXTL2MGFCSRTPDDPAYXEYLMISPP5AWY337MHX7HFLQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.449325822 (291h21m27.000796283s) [noise=true , address=185.167.204.205:4031 , piece_id=ZEI2YSIQC655R7WS43WADXGFT5RKT5LFSN6IZEQ6SHTDVOV7BZBQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.449761211 (291h21m27.001231672s) [noise=true , address=185.167.204.205:4018 , piece_id=7OSLPWJLKFTIJPZEAFMZOAH6JAAMX6HKROI25FODFYBPNJWSMLWA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.450419845 (291h21m27.001890306s) [noise=true , address=185.167.204.206:4062 , piece_id=UF3S6NAVGS2KQJPGA5WXP7YG4RQO44P445SNPCWXKXMONZAYUTNQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.453201252 (291h21m27.004671713s) [noise=true , address=185.167.204.205:4007 , piece_id=MEBGOPZ6MFMQY4UZARS2LKJKEMNN5YGSLKPJGTWSKS45CUNXSRGQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:29.661209626 (291h21m28.212680087s) [noise=true , address=89.149.201.147:4016 , piece_id=TGRWGGGBKDANC7YKROWPRMIYNAVFEVE3RXOSUOINFI7KTRS2IYUQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:29.693169182 (291h21m28.244639643s) [noise=true , address=89.149.201.147:4002 , piece_id=UIYJSCMY3LCLRHZUJB6JFJHHXKMYTVKERTZW7F7X44MWJETPY6XA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:29.721204497 (291h21m28.272674958s) [noise=true , address=89.149.201.147:4008 , piece_id=HCX7I4ICPEHDWTYDFIQMKZ5EE5CGPQLYAIHF7WZMTUUU6CNZPHTA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:29.764584711 (291h21m28.316055172s) [noise=true , address=89.149.201.147:4001 , piece_id=QUS6NB3NQBLJFB6YCK3EBDPKCTWEWGKAT3NOR24TVI65SQKHWDOQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:29.768284075 (291h21m28.319754536s) [noise=true , address=89.149.201.147:4011 , piece_id=ZATNMD5HOK35AWGJJ5V66VDJ4YGUVZQJ2OBERQL32ZV5GW2HBSMQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:29.789681961 (291h21m28.341152422s) [noise=true , address=89.149.201.147:4003 , piece_id=RUAYGN2RGF5JJTWFEGIHWH3BHEUDSS5DN2QJ6YLSHPF5BGUXD3RA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:37.298598682 (291h21m35.850069143s) [noise=true , address=81.171.14.70:4043 , piece_id=T3NF6JR3VJIGDMPNUBLMJ3AOTQP5IHOGJYORMCSS7ZMOENQETZ7A, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:37.497478356 (291h21m36.048948818s) [noise=true , address=81.171.14.70:4032 , piece_id=CFH2RTR3BLBUB5ZIZCFC6UHT3FHT3B2XSCHQMKXNGDJJGPHEMCTQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:40.406166933 (291h21m38.957637394s) [noise=true , address=81.171.14.70:4045 , piece_id=BRWQ7G3E547PJE2RLGSESIB3HMP5C4M5NSKAJNGWSFXUBEO6SYEQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:42.511041157 (291h21m41.062511608s) [noise=true , address=81.171.14.70:4029 , piece_id=DKJM5E7EY7LOSBSEBQOZRC6CI5ZPPE47HFOFQJROPCUMUV567CAA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:26:12.012682521 (291h22m10.564152982s) [noise=true , address=81.171.14.70:4042 , piece_id=XNREZEUZ5L5KZ3L7ZOL6IRK7IHU6DJN4RLEQZRFMDKDH2JWC24CQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:26:15.596046283 (291h22m14.147516744s) [noise=true , address=81.171.14.70:4033 , piece_id=JRKPD4EP6FZM5HISOIDJYZY7FAWWQCADAS47FGW2JM5A7RFJR2BQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:26:18.453059659 (291h22m17.00453012s) [noise=true , address=81.171.14.70:4046 , piece_id=OSONRSF6KJS7F2QJVCC3FPZK4AV7QPG5YSM47CW5LIW7EME44AWA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:26:23.442583344 (291h22m21.994053805s) [noise=true , address=81.171.14.70:4034 , piece_id=ITRLKYVNFSYBLABNGNFO3HRJNMX3MWB345FIMWSIPBNAVZPA47UA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:26:23.442621967 (291h22m21.994092428s) [seg_pos={0 0}, upload=256586]: Segment reached optimal threshold of 26 pieces. Nov 21 11:26:23.442836976 (291h22m21.994307437s) [noise=true , address=81.171.14.70:4044 , piece_id=QYUBNGS2FAFHIWMW7GAGYJ3UVIWVP62LIDCOK6QUXEIQNIXRJDAQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1MUVGecysnTSFcJoHUL7QMWb3h6dtXgUd7FbA672fQCyEWhePr): protocol: expected piece hash; context canceled; EOF Nov 21 11:26:23.442845934 (291h22m21.994316395s) [noise=true , address=81.171.14.70:4044 , piece_id=QYUBNGS2FAFHIWMW7GAGYJ3UVIWVP62LIDCOK6QUXEIQNIXRJDAQ, seg_pos={0 0}, upload=256586]: Long tail cancellation due to optimal threshold Nov 21 11:26:23.443227491 (291h22m21.994697942s) [noise=true , address=81.171.14.70:4027 , piece_id=4FDFCCWF3EIROQAFLOHGS743523OYGSXWQSPQEVD3AMQ6WKDQI7Q, seg_pos={0 0}, upload=256586]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122cYvgTxSvC2aEXnBkRk2o8ARTBXj9gqPoa12vW8hogSN6kpA4): protocol: expected piece hash; context canceled; EOF Nov 21 11:26:23.443237770 (291h22m21.994708231s) [noise=true , address=81.171.14.70:4027 , piece_id=4FDFCCWF3EIROQAFLOHGS743523OYGSXWQSPQEVD3AMQ6WKDQI7Q, seg_pos={0 0}, upload=256586]: Long tail cancellation due to optimal threshold Nov 21 11:26:23.443274720 (291h22m21.994745181s) [seg_pos={0 0}, upload=256586]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Nov 21 11:26:23.443378929 (291h22m21.99484939s) [upload=256586]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 21 11:26:23.443387044 (291h22m21.994857495s) [upload=256586]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 21 11:26:23.685139626 (291h22m22.236610077s) [upload=256586]: Done uploading segments... Nov 21 11:26:23.685160676 (291h22m22.236631127s) [upload=256586]: Upload finished. err: <nil> Duration: 6.492744779s core uplink eu-select-1 eu1-placement-15
checkers.HTTP{"GET" "https://link.storjshare.io/raw/jw7f74fzbtwblz3mwqolmnivikxq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.us1.linksharing) antiblip: 1 out of 60 is failed (threshold=6) [X...........................................................] duration: 260.71472ms remote: "tcp:207.211.208.129:443" target: https://link.storjshare.io/raw/jw7f74fzbtwblz3mwqolmnivikxq/uptime-checks/hello-world.txt trace: https://jaeger.dev.storj.io/trace/41cfd75a627799ce FAIL (2025-11-21 11:47:02 UTC) orbiter warning: Get "https://link.storjshare.io/raw/jw7f74fzbtwblz3mwqolmnivikxq/uptime-checks/hello-world.txt": context deadline exceeded Duration: 260.757712ms edge linksharing bgp us1
checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us1.gateway) antiblip: 1 out of 60 is failed (threshold=3) [...X........................................................] duration: 261.986521ms remote: "tcp:207.211.208.129:443" target: https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222 trace: https://jaeger.dev.storj.io/trace/3e5c193367f9a974 FAIL (2025-11-21 11:47:02 UTC) orbiter warning: Get "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222": context deadline exceeded Duration: 262.018022ms edge gateway vhost bgp us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 47 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.............] duration: 3.474874461s trace: https://jaeger.dev.storj.io/trace/6af15a3a67b655d5 FAIL (2025-11-21 11:47:02 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 3.474898206s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [....X...X.X......X.XX...X.X.X....X............X....X.X.X..X.] duration: 6.495331857s trace: https://jaeger.dev.storj.io/trace/4b54cc8c3329136a FAIL (2025-11-21 11:46:41 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf, Piece ID: VEEKWGN3WKJHH3NWQV6BSHONOMIX77RBH36HCSY6CAVOWL3H4XNA) context deadline exceeded; piecestore: (Node ID: 129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, Piece ID: 2OENXXCJZPACMD4HHGQYNMLGPIUMXVNJ4TA46NKEV6H57IUFAEPQ) context deadline exceeded ======LOGS====== atchRequestItem_ObjectBegin Nov 21 11:46:05.218076332 (291h42m3.769546793s) [upload=256885]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 21 11:46:05.463459889 (291h42m4.01493035s) [seg_pos={0 0}, upload=256885]: Begin upload segment... Nov 21 11:46:05.463529121 (291h42m4.014999582s) [seg_pos={0 0}, upload=256885]: Done begin upload segment. Nov 21 11:46:05.463569097 (291h42m4.015039568s) [noise=true , address=77.90.31.10:4517 , piece_id=2OENXXCJZPACMD4HHGQYNMLGPIUMXVNJ4TA46NKEV6H57IUFAEPQ, seg_pos={0 0}, upload=256885]: Uploading piece... Nov 21 11:46:05.463682352 (291h42m4.015152814s) [noise=true , address=145.79.227.32:4834 , piece_id=K3BZY6ELVS7RQ52OW32GA726RLCUJTCW7GTNXV4HJ4NNMV4XQADA, seg_pos={0 0}, upload=256885]: Uploading piece... Nov 21 11:46:05.464677999 (291h42m4.01614846s) [noise=true , address=145.79.227.21:4473 , piece_id=I77YPBHWFISIPPWKVZLW7XLZEATURQPCN2DW3SW5XI6ZXQSA2O6A, seg_pos={0 0}, upload=256885]: Uploading piece... Nov 21 11:46:05.464875194 (291h42m4.016345655s) [noise=true , address=145.79.227.16:4912 , piece_id=VM2V3WCD3L7CAHQBO7CLLF3PUYYXET37PGTDK3DSV3S26TMNQALA, seg_pos={0 0}, upload=256885]: Uploading piece... Nov 21 11:46:05.465336483 (291h42m4.016806934s) [noise=true , address=145.79.227.20:4438 , piece_id=QWE5LK5I5VLCRJCWZUXQGFCSAOTETWPANZHHDBJ7J7DF6LU5UKGA, seg_pos={0 0}, upload=256885]: Uploading piece... Nov 21 11:46:05.465677742 (291h42m4.017148203s) [noise=true , address=145.79.227.23:4546 , piece_id=JTA43DTWCHYDTYPZXEDL5CVXSMAF4IL2HISPYFGNOTAQ6XY5XQ2A, seg_pos={0 0}, upload=256885]: Uploading piece... Nov 21 11:46:05.466539904 (291h42m4.018010365s) [noise=true , address=145.79.227.14:4837 , piece_id=HDMJET6QEHKFK3PYDB6GAIVARJ4CRNTBLVPIKUU3MEHSHR547NXA, seg_pos={0 0}, upload=256885]: Uploading piece... Nov 21 11:46:05.467003296 (291h42m4.018473757s) [noise=true , address=45.91.116.11:4119 , piece_id=L6GUBNFI5W3ZI57EF5ZHWRZ6CAAFBWLZ5AJDAMLROZS4IUZN2XAQ, seg_pos={0 0}, upload=256885]: Uploading piece... Nov 21 11:46:05.467330299 (291h42m4.01880076s) [noise=true , address=145.79.227.30:4763 , piece_id=YEDH7X4AXG2JI2R7AQJMC27VLRLJUE4PS77KNESUTVNT7GXTVPOQ, seg_pos={0 0}, upload=256885]: Uploading piece... Nov 21 11:46:05.469087775 (291h42m4.020558236s) [noise=true , address=145.79.227.26:4653 , piece_id=DKE2CLJKK5SIVUAA44H3DWAU4XMZKAMOQQNPH5DFAOYEPTE4D7JQ, seg_pos={0 0}, upload=256885]: Uploading piece... Nov 21 11:46:05.469425679 (291h42m4.02089614s) [noise=true , address=45.91.116.7:4011 , piece_id=VEEKWGN3WKJHH3NWQV6BSHONOMIX77RBH36HCSY6CAVOWL3H4XNA, seg_pos={0 0}, upload=256885]: Uploading piece... Nov 21 11:46:05.471708516 (291h42m4.023178977s) [noise=true , address=145.79.227.15:4876 , piece_id=7B33KVRRMEIMKQ2WLADFHOUIOWHY572LIMHHCAV7T6FIZYE7NEKA, seg_pos={0 0}, upload=256885]: Uploading piece... Nov 21 11:46:05.473158827 (291h42m4.024629288s) [noise=true , address=145.79.227.28:4725 , piece_id=QIZRSZLEGKGGLTWSQBDF3PDWAL7FHXXM2OE4SYGAS2Q7VL6ZWRHQ, seg_pos={0 0}, upload=256885]: Uploading piece... Nov 21 11:46:05.474179401 (291h42m4.025649862s) [noise=true , address=45.91.116.10:4081 , piece_id=MBNARROM63E25EVCDA3KV2NERCXLKEMXJTQJYFPHFI7PQALGRN4A, seg_pos={0 0}, upload=256885]: Uploading piece... Nov 21 11:46:05.476012201 (291h42m4.027482662s) [noise=true , address=77.90.31.13:4627 , piece_id=YISCR23XNGEJR7GSRGIYOM3PSCYRC6I4TXRMSZHYCF4WWSDW5YAA, seg_pos={0 0}, upload=256885]: Uploading piece... Nov 21 11:46:05.477130861 (291h42m4.028601322s) [noise=true , address=45.91.116.12:4225 , piece_id=T7DRRQ7VZGKCCASCBOMAKZF2DL6SCGP4R32I4PFKCTWSTQAGA6VQ, seg_pos={0 0}, upload=256885]: Uploading piece... Nov 21 11:46:07.070951748 (291h42m5.622422199s) [upload=256885]: Next returned nil segment Nov 21 11:46:07.070964212 (291h42m5.622434673s) [upload=256885]: Waiting for error group managing segments... Nov 21 11:46:07.453777160 (291h42m6.005247622s) [noise=true , address=145.79.227.15:4876 , piece_id=7B33KVRRMEIMKQ2WLADFHOUIOWHY572LIMHHCAV7T6FIZYE7NEKA, seg_pos={0 0}, upload=256885]: Done uploading piece. err: <nil> Nov 21 11:46:07.471492834 (291h42m6.022963285s) [noise=true , address=145.79.227.32:4834 , piece_id=K3BZY6ELVS7RQ52OW32GA726RLCUJTCW7GTNXV4HJ4NNMV4XQADA, seg_pos={0 0}, upload=256885]: Done uploading piece. err: <nil> Nov 21 11:46:07.479408678 (291h42m6.030879139s) [noise=true , address=145.79.227.21:4473 , piece_id=I77YPBHWFISIPPWKVZLW7XLZEATURQPCN2DW3SW5XI6ZXQSA2O6A, seg_pos={0 0}, upload=256885]: Done uploading piece. err: <nil> Nov 21 11:46:07.480651514 (291h42m6.032121975s) [noise=true , address=145.79.227.14:4837 , piece_id=HDMJET6QEHKFK3PYDB6GAIVARJ4CRNTBLVPIKUU3MEHSHR547NXA, seg_pos={0 0}, upload=256885]: Done uploading piece. err: <nil> Nov 21 11:46:07.480767756 (291h42m6.032238217s) [noise=true , address=145.79.227.23:4546 , piece_id=JTA43DTWCHYDTYPZXEDL5CVXSMAF4IL2HISPYFGNOTAQ6XY5XQ2A, seg_pos={0 0}, upload=256885]: Done uploading piece. err: <nil> Nov 21 11:46:07.482437535 (291h42m6.033907997s) [noise=true , address=145.79.227.16:4912 , piece_id=VM2V3WCD3L7CAHQBO7CLLF3PUYYXET37PGTDK3DSV3S26TMNQALA, seg_pos={0 0}, upload=256885]: Done uploading piece. err: <nil> Nov 21 11:46:07.483056504 (291h42m6.034526965s) [noise=true , address=145.79.227.28:4725 , piece_id=QIZRSZLEGKGGLTWSQBDF3PDWAL7FHXXM2OE4SYGAS2Q7VL6ZWRHQ, seg_pos={0 0}, upload=256885]: Done uploading piece. err: <nil> Nov 21 11:46:07.487216796 (291h42m6.038687257s) [noise=true , address=145.79.227.20:4438 , piece_id=QWE5LK5I5VLCRJCWZUXQGFCSAOTETWPANZHHDBJ7J7DF6LU5UKGA, seg_pos={0 0}, upload=256885]: Done uploading piece. err: <nil> Nov 21 11:46:07.489358544 (291h42m6.040829005s) [noise=true , address=145.79.227.30:4763 , piece_id=YEDH7X4AXG2JI2R7AQJMC27VLRLJUE4PS77KNESUTVNT7GXTVPOQ, seg_pos={0 0}, upload=256885]: Done uploading piece. err: <nil> Nov 21 11:46:07.495585021 (291h42m6.047055482s) [noise=true , address=145.79.227.26:4653 , piece_id=DKE2CLJKK5SIVUAA44H3DWAU4XMZKAMOQQNPH5DFAOYEPTE4D7JQ, seg_pos={0 0}, upload=256885]: Done uploading piece. err: <nil> Nov 21 11:46:08.107882821 (291h42m6.659353272s) [noise=true , address=45.91.116.11:4119 , piece_id=L6GUBNFI5W3ZI57EF5ZHWRZ6CAAFBWLZ5AJDAMLROZS4IUZN2XAQ, seg_pos={0 0}, upload=256885]: Done uploading piece. err: <nil> Nov 21 11:46:08.138116502 (291h42m6.689586953s) [noise=true , address=77.90.31.10:4517 , piece_id=2OENXXCJZPACMD4HHGQYNMLGPIUMXVNJ4TA46NKEV6H57IUFAEPQ, seg_pos={0 0}, upload=256885]: Done uploading piece. err: <nil> Nov 21 11:46:08.277302248 (291h42m6.828772709s) [noise=true , address=45.91.116.7:4011 , piece_id=VEEKWGN3WKJHH3NWQV6BSHONOMIX77RBH36HCSY6CAVOWL3H4XNA, seg_pos={0 0}, upload=256885]: Done uploading piece. err: <nil> Nov 21 11:46:08.277329250 (291h42m6.828799701s) [seg_pos={0 0}, upload=256885]: Segment reached optimal threshold of 13 pieces. Nov 21 11:46:08.277556041 (291h42m6.829026502s) [noise=true , address=77.90.31.13:4627 , piece_id=YISCR23XNGEJR7GSRGIYOM3PSCYRC6I4TXRMSZHYCF4WWSDW5YAA, seg_pos={0 0}, upload=256885]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB): protocol: write tcp 188.166.225.170:43890->77.90.31.13:4627: use of closed network connection; write tcp 188.166.225.170:43890->77.90.31.13:4627: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43890->77.90.31.13:4627: use of closed network connection Nov 21 11:46:08.277565108 (291h42m6.829035579s) [noise=true , address=77.90.31.13:4627 , piece_id=YISCR23XNGEJR7GSRGIYOM3PSCYRC6I4TXRMSZHYCF4WWSDW5YAA, seg_pos={0 0}, upload=256885]: Long tail cancellation due to optimal threshold Nov 21 11:46:08.277737396 (291h42m6.829207857s) [noise=true , address=45.91.116.10:4081 , piece_id=MBNARROM63E25EVCDA3KV2NERCXLKEMXJTQJYFPHFI7PQALGRN4A, seg_pos={0 0}, upload=256885]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG): protocol: write tcp 188.166.225.170:34024->45.91.116.10:4081: use of closed network connection; write tcp 188.166.225.170:34024->45.91.116.10:4081: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34024->45.91.116.10:4081: use of closed network connection Nov 21 11:46:08.277746264 (291h42m6.829216725s) [noise=true , address=45.91.116.10:4081 , piece_id=MBNARROM63E25EVCDA3KV2NERCXLKEMXJTQJYFPHFI7PQALGRN4A, seg_pos={0 0}, upload=256885]: Long tail cancellation due to optimal threshold Nov 21 11:46:08.278096992 (291h42m6.829567453s) [noise=true , address=45.91.116.12:4225 , piece_id=T7DRRQ7VZGKCCASCBOMAKZF2DL6SCGP4R32I4PFKCTWSTQAGA6VQ, seg_pos={0 0}, upload=256885]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: write tcp 188.166.225.170:54264->45.91.116.12:4225: use of closed network connection; write tcp 188.166.225.170:54264->45.91.116.12:4225: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54264->45.91.116.12:4225: use of closed network connection Nov 21 11:46:08.278117781 (291h42m6.829588242s) [noise=true , address=45.91.116.12:4225 , piece_id=T7DRRQ7VZGKCCASCBOMAKZF2DL6SCGP4R32I4PFKCTWSTQAGA6VQ, seg_pos={0 0}, upload=256885]: Long tail cancellation due to optimal threshold Nov 21 11:46:08.278148319 (291h42m6.82961878s) [seg_pos={0 0}, upload=256885]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 21 11:46:08.278277595 (291h42m6.829748046s) [upload=256885]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 21 11:46:08.278285380 (291h42m6.829755841s) [upload=256885]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 21 11:46:08.568224629 (291h42m7.11969509s) [upload=256885]: Done uploading segments... Nov 21 11:46:08.568453836 (291h42m7.119924297s) [upload=256885]: Upload finished. err: <nil> Duration:* 6.495363898s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 48 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX............] duration: 3.411085855s trace: https://jaeger.dev.storj.io/trace/6af15a3a67b655d5 FAIL (2025-11-21 11:46:36 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 3.411116484s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [.....X...X.X......X.XX...X.X.X....X............X....X.X.X..X] duration: 30.004533823s trace: https://jaeger.dev.storj.io/trace/4b54cc8c3329136a FAIL (2025-11-21 11:46:34 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf, Piece ID: VEEKWGN3WKJHH3NWQV6BSHONOMIX77RBH36HCSY6CAVOWL3H4XNA) context deadline exceeded; piecestore: (Node ID: 129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, Piece ID: 2OENXXCJZPACMD4HHGQYNMLGPIUMXVNJ4TA46NKEV6H57IUFAEPQ) context deadline exceeded ======LOGS====== atchRequestItem_ObjectBegin Nov 21 11:46:05.218076332 (291h42m3.769546793s) [upload=256885]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 21 11:46:05.463459889 (291h42m4.01493035s) [seg_pos={0 0}, upload=256885]: Begin upload segment... Nov 21 11:46:05.463529121 (291h42m4.014999582s) [seg_pos={0 0}, upload=256885]: Done begin upload segment. Nov 21 11:46:05.463569097 (291h42m4.015039568s) [noise=true , address=77.90.31.10:4517 , piece_id=2OENXXCJZPACMD4HHGQYNMLGPIUMXVNJ4TA46NKEV6H57IUFAEPQ, seg_pos={0 0}, upload=256885]: Uploading piece... Nov 21 11:46:05.463682352 (291h42m4.015152814s) [noise=true , address=145.79.227.32:4834 , piece_id=K3BZY6ELVS7RQ52OW32GA726RLCUJTCW7GTNXV4HJ4NNMV4XQADA, seg_pos={0 0}, upload=256885]: Uploading piece... Nov 21 11:46:05.464677999 (291h42m4.01614846s) [noise=true , address=145.79.227.21:4473 , piece_id=I77YPBHWFISIPPWKVZLW7XLZEATURQPCN2DW3SW5XI6ZXQSA2O6A, seg_pos={0 0}, upload=256885]: Uploading piece... Nov 21 11:46:05.464875194 (291h42m4.016345655s) [noise=true , address=145.79.227.16:4912 , piece_id=VM2V3WCD3L7CAHQBO7CLLF3PUYYXET37PGTDK3DSV3S26TMNQALA, seg_pos={0 0}, upload=256885]: Uploading piece... Nov 21 11:46:05.465336483 (291h42m4.016806934s) [noise=true , address=145.79.227.20:4438 , piece_id=QWE5LK5I5VLCRJCWZUXQGFCSAOTETWPANZHHDBJ7J7DF6LU5UKGA, seg_pos={0 0}, upload=256885]: Uploading piece... Nov 21 11:46:05.465677742 (291h42m4.017148203s) [noise=true , address=145.79.227.23:4546 , piece_id=JTA43DTWCHYDTYPZXEDL5CVXSMAF4IL2HISPYFGNOTAQ6XY5XQ2A, seg_pos={0 0}, upload=256885]: Uploading piece... Nov 21 11:46:05.466539904 (291h42m4.018010365s) [noise=true , address=145.79.227.14:4837 , piece_id=HDMJET6QEHKFK3PYDB6GAIVARJ4CRNTBLVPIKUU3MEHSHR547NXA, seg_pos={0 0}, upload=256885]: Uploading piece... Nov 21 11:46:05.467003296 (291h42m4.018473757s) [noise=true , address=45.91.116.11:4119 , piece_id=L6GUBNFI5W3ZI57EF5ZHWRZ6CAAFBWLZ5AJDAMLROZS4IUZN2XAQ, seg_pos={0 0}, upload=256885]: Uploading piece... Nov 21 11:46:05.467330299 (291h42m4.01880076s) [noise=true , address=145.79.227.30:4763 , piece_id=YEDH7X4AXG2JI2R7AQJMC27VLRLJUE4PS77KNESUTVNT7GXTVPOQ, seg_pos={0 0}, upload=256885]: Uploading piece... Nov 21 11:46:05.469087775 (291h42m4.020558236s) [noise=true , address=145.79.227.26:4653 , piece_id=DKE2CLJKK5SIVUAA44H3DWAU4XMZKAMOQQNPH5DFAOYEPTE4D7JQ, seg_pos={0 0}, upload=256885]: Uploading piece... Nov 21 11:46:05.469425679 (291h42m4.02089614s) [noise=true , address=45.91.116.7:4011 , piece_id=VEEKWGN3WKJHH3NWQV6BSHONOMIX77RBH36HCSY6CAVOWL3H4XNA, seg_pos={0 0}, upload=256885]: Uploading piece... Nov 21 11:46:05.471708516 (291h42m4.023178977s) [noise=true , address=145.79.227.15:4876 , piece_id=7B33KVRRMEIMKQ2WLADFHOUIOWHY572LIMHHCAV7T6FIZYE7NEKA, seg_pos={0 0}, upload=256885]: Uploading piece... Nov 21 11:46:05.473158827 (291h42m4.024629288s) [noise=true , address=145.79.227.28:4725 , piece_id=QIZRSZLEGKGGLTWSQBDF3PDWAL7FHXXM2OE4SYGAS2Q7VL6ZWRHQ, seg_pos={0 0}, upload=256885]: Uploading piece... Nov 21 11:46:05.474179401 (291h42m4.025649862s) [noise=true , address=45.91.116.10:4081 , piece_id=MBNARROM63E25EVCDA3KV2NERCXLKEMXJTQJYFPHFI7PQALGRN4A, seg_pos={0 0}, upload=256885]: Uploading piece... Nov 21 11:46:05.476012201 (291h42m4.027482662s) [noise=true , address=77.90.31.13:4627 , piece_id=YISCR23XNGEJR7GSRGIYOM3PSCYRC6I4TXRMSZHYCF4WWSDW5YAA, seg_pos={0 0}, upload=256885]: Uploading piece... Nov 21 11:46:05.477130861 (291h42m4.028601322s) [noise=true , address=45.91.116.12:4225 , piece_id=T7DRRQ7VZGKCCASCBOMAKZF2DL6SCGP4R32I4PFKCTWSTQAGA6VQ, seg_pos={0 0}, upload=256885]: Uploading piece... Nov 21 11:46:07.070951748 (291h42m5.622422199s) [upload=256885]: Next returned nil segment Nov 21 11:46:07.070964212 (291h42m5.622434673s) [upload=256885]: Waiting for error group managing segments... Nov 21 11:46:07.453777160 (291h42m6.005247622s) [noise=true , address=145.79.227.15:4876 , piece_id=7B33KVRRMEIMKQ2WLADFHOUIOWHY572LIMHHCAV7T6FIZYE7NEKA, seg_pos={0 0}, upload=256885]: Done uploading piece. err: <nil> Nov 21 11:46:07.471492834 (291h42m6.022963285s) [noise=true , address=145.79.227.32:4834 , piece_id=K3BZY6ELVS7RQ52OW32GA726RLCUJTCW7GTNXV4HJ4NNMV4XQADA, seg_pos={0 0}, upload=256885]: Done uploading piece. err: <nil> Nov 21 11:46:07.479408678 (291h42m6.030879139s) [noise=true , address=145.79.227.21:4473 , piece_id=I77YPBHWFISIPPWKVZLW7XLZEATURQPCN2DW3SW5XI6ZXQSA2O6A, seg_pos={0 0}, upload=256885]: Done uploading piece. err: <nil> Nov 21 11:46:07.480651514 (291h42m6.032121975s) [noise=true , address=145.79.227.14:4837 , piece_id=HDMJET6QEHKFK3PYDB6GAIVARJ4CRNTBLVPIKUU3MEHSHR547NXA, seg_pos={0 0}, upload=256885]: Done uploading piece. err: <nil> Nov 21 11:46:07.480767756 (291h42m6.032238217s) [noise=true , address=145.79.227.23:4546 , piece_id=JTA43DTWCHYDTYPZXEDL5CVXSMAF4IL2HISPYFGNOTAQ6XY5XQ2A, seg_pos={0 0}, upload=256885]: Done uploading piece. err: <nil> Nov 21 11:46:07.482437535 (291h42m6.033907997s) [noise=true , address=145.79.227.16:4912 , piece_id=VM2V3WCD3L7CAHQBO7CLLF3PUYYXET37PGTDK3DSV3S26TMNQALA, seg_pos={0 0}, upload=256885]: Done uploading piece. err: <nil> Nov 21 11:46:07.483056504 (291h42m6.034526965s) [noise=true , address=145.79.227.28:4725 , piece_id=QIZRSZLEGKGGLTWSQBDF3PDWAL7FHXXM2OE4SYGAS2Q7VL6ZWRHQ, seg_pos={0 0}, upload=256885]: Done uploading piece. err: <nil> Nov 21 11:46:07.487216796 (291h42m6.038687257s) [noise=true , address=145.79.227.20:4438 , piece_id=QWE5LK5I5VLCRJCWZUXQGFCSAOTETWPANZHHDBJ7J7DF6LU5UKGA, seg_pos={0 0}, upload=256885]: Done uploading piece. err: <nil> Nov 21 11:46:07.489358544 (291h42m6.040829005s) [noise=true , address=145.79.227.30:4763 , piece_id=YEDH7X4AXG2JI2R7AQJMC27VLRLJUE4PS77KNESUTVNT7GXTVPOQ, seg_pos={0 0}, upload=256885]: Done uploading piece. err: <nil> Nov 21 11:46:07.495585021 (291h42m6.047055482s) [noise=true , address=145.79.227.26:4653 , piece_id=DKE2CLJKK5SIVUAA44H3DWAU4XMZKAMOQQNPH5DFAOYEPTE4D7JQ, seg_pos={0 0}, upload=256885]: Done uploading piece. err: <nil> Nov 21 11:46:08.107882821 (291h42m6.659353272s) [noise=true , address=45.91.116.11:4119 , piece_id=L6GUBNFI5W3ZI57EF5ZHWRZ6CAAFBWLZ5AJDAMLROZS4IUZN2XAQ, seg_pos={0 0}, upload=256885]: Done uploading piece. err: <nil> Nov 21 11:46:08.138116502 (291h42m6.689586953s) [noise=true , address=77.90.31.10:4517 , piece_id=2OENXXCJZPACMD4HHGQYNMLGPIUMXVNJ4TA46NKEV6H57IUFAEPQ, seg_pos={0 0}, upload=256885]: Done uploading piece. err: <nil> Nov 21 11:46:08.277302248 (291h42m6.828772709s) [noise=true , address=45.91.116.7:4011 , piece_id=VEEKWGN3WKJHH3NWQV6BSHONOMIX77RBH36HCSY6CAVOWL3H4XNA, seg_pos={0 0}, upload=256885]: Done uploading piece. err: <nil> Nov 21 11:46:08.277329250 (291h42m6.828799701s) [seg_pos={0 0}, upload=256885]: Segment reached optimal threshold of 13 pieces. Nov 21 11:46:08.277556041 (291h42m6.829026502s) [noise=true , address=77.90.31.13:4627 , piece_id=YISCR23XNGEJR7GSRGIYOM3PSCYRC6I4TXRMSZHYCF4WWSDW5YAA, seg_pos={0 0}, upload=256885]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB): protocol: write tcp 188.166.225.170:43890->77.90.31.13:4627: use of closed network connection; write tcp 188.166.225.170:43890->77.90.31.13:4627: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43890->77.90.31.13:4627: use of closed network connection Nov 21 11:46:08.277565108 (291h42m6.829035579s) [noise=true , address=77.90.31.13:4627 , piece_id=YISCR23XNGEJR7GSRGIYOM3PSCYRC6I4TXRMSZHYCF4WWSDW5YAA, seg_pos={0 0}, upload=256885]: Long tail cancellation due to optimal threshold Nov 21 11:46:08.277737396 (291h42m6.829207857s) [noise=true , address=45.91.116.10:4081 , piece_id=MBNARROM63E25EVCDA3KV2NERCXLKEMXJTQJYFPHFI7PQALGRN4A, seg_pos={0 0}, upload=256885]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG): protocol: write tcp 188.166.225.170:34024->45.91.116.10:4081: use of closed network connection; write tcp 188.166.225.170:34024->45.91.116.10:4081: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34024->45.91.116.10:4081: use of closed network connection Nov 21 11:46:08.277746264 (291h42m6.829216725s) [noise=true , address=45.91.116.10:4081 , piece_id=MBNARROM63E25EVCDA3KV2NERCXLKEMXJTQJYFPHFI7PQALGRN4A, seg_pos={0 0}, upload=256885]: Long tail cancellation due to optimal threshold Nov 21 11:46:08.278096992 (291h42m6.829567453s) [noise=true , address=45.91.116.12:4225 , piece_id=T7DRRQ7VZGKCCASCBOMAKZF2DL6SCGP4R32I4PFKCTWSTQAGA6VQ, seg_pos={0 0}, upload=256885]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: write tcp 188.166.225.170:54264->45.91.116.12:4225: use of closed network connection; write tcp 188.166.225.170:54264->45.91.116.12:4225: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54264->45.91.116.12:4225: use of closed network connection Nov 21 11:46:08.278117781 (291h42m6.829588242s) [noise=true , address=45.91.116.12:4225 , piece_id=T7DRRQ7VZGKCCASCBOMAKZF2DL6SCGP4R32I4PFKCTWSTQAGA6VQ, seg_pos={0 0}, upload=256885]: Long tail cancellation due to optimal threshold Nov 21 11:46:08.278148319 (291h42m6.82961878s) [seg_pos={0 0}, upload=256885]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 21 11:46:08.278277595 (291h42m6.829748046s) [upload=256885]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 21 11:46:08.278285380 (291h42m6.829755841s) [upload=256885]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 21 11:46:08.568224629 (291h42m7.11969509s) [upload=256885]: Done uploading segments... Nov 21 11:46:08.568453836 (291h42m7.119924297s) [upload=256885]: Upload finished. err: <nil> Duration:* 30.004554261s core uplink us-select-1
checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us1.gateway) antiblip: 1 out of 60 is failed (threshold=3) [....X.......................................................] duration: 254.994379ms remote: "tcp:207.211.208.129:443" target: https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222 trace: https://jaeger.dev.storj.io/trace/3e5c193367f9a974 FAIL (2025-11-21 11:46:32 UTC) orbiter warning: Get "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222": context deadline exceeded Duration: 255.019015ms edge gateway vhost bgp us1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 2 out of 60 is failed (threshold=3) [..........................X.X...............................] duration: 5.207179722s trace: https://jaeger.dev.storj.io/trace/a7f69ee106efb7f FAIL (2025-11-21 11:46:23 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== upload=256586]: Uploading piece... Nov 21 11:25:27.210346024 (291h21m25.761816485s) [noise=true , address=89.149.201.147:4008 , piece_id=HCX7I4ICPEHDWTYDFIQMKZ5EE5CGPQLYAIHF7WZMTUUU6CNZPHTA, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.211425289 (291h21m25.76289575s) [noise=true , address=81.171.14.70:4044 , piece_id=QYUBNGS2FAFHIWMW7GAGYJ3UVIWVP62LIDCOK6QUXEIQNIXRJDAQ, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.213103795 (291h21m25.764574246s) [noise=true , address=185.167.204.205:4022 , piece_id=BES3YK6PYQV2AKOWSAGHF7DGGWGWSWPIJU2ICAQ5WBHKZ5VT5L6A, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.213408395 (291h21m25.764878856s) [noise=true , address=185.167.204.205:4027 , piece_id=IVPONKBF2PIHZQAW3L4YA4HOMQV5BQML6SKKCJLIEGT5SZSQ2AOA, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.214501747 (291h21m25.765972208s) [noise=true , address=81.171.14.70:4042 , piece_id=XNREZEUZ5L5KZ3L7ZOL6IRK7IHU6DJN4RLEQZRFMDKDH2JWC24CQ, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.215013752 (291h21m25.766484213s) [noise=true , address=81.171.14.70:4034 , piece_id=ITRLKYVNFSYBLABNGNFO3HRJNMX3MWB345FIMWSIPBNAVZPA47UA, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.215087572 (291h21m25.766558044s) [noise=true , address=185.167.204.205:4031 , piece_id=ZEI2YSIQC655R7WS43WADXGFT5RKT5LFSN6IZEQ6SHTDVOV7BZBQ, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.216432844 (291h21m25.767903305s) [noise=true , address=185.167.204.206:4056 , piece_id=EAMZIK6AVJL7IMFF5VOLW3SMCGYKSKB2VXJTNUNLJCVIT7EKXNMQ, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.216662281 (291h21m25.768132742s) [noise=true , address=89.149.201.147:4011 , piece_id=ZATNMD5HOK35AWGJJ5V66VDJ4YGUVZQJ2OBERQL32ZV5GW2HBSMQ, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.218008314 (291h21m25.769478775s) [noise=true , address=81.171.14.70:4032 , piece_id=CFH2RTR3BLBUB5ZIZCFC6UHT3FHT3B2XSCHQMKXNGDJJGPHEMCTQ, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:28.035150976 (291h21m26.586621437s) [upload=256586]: Next returned nil segment Nov 21 11:25:28.035169622 (291h21m26.586640083s) [upload=256586]: Waiting for error group managing segments... Nov 21 11:25:28.416646836 (291h21m26.968117317s) [noise=true , address=185.167.204.206:4064 , piece_id=SRFIWFQLRAGCNIKYLHMGT4QRUL3WH2QPLLIKAJ7DLFJEQUW5DPPA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.417338854 (291h21m26.968809325s) [noise=true , address=185.167.204.205:4033 , piece_id=FEWQVHPIKN3Q6O5AZ6AGJBLUPZ34PBZFB5X3XNSKGPOGI4AIO7JA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.421089406 (291h21m26.972559867s) [noise=true , address=185.167.204.206:4072 , piece_id=GYCO53OWGOMOHK4FHC5GENN4LTQGSZKZKOI5V534RLV3LHLPKC6Q, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.426252887 (291h21m26.977723348s) [noise=true , address=185.167.204.206:4046 , piece_id=BZ57ZA35ITEOA6CAZJGNODK5MUTXEZWDV7E2OGBZOSMS4OQZYAEA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.431720689 (291h21m26.98319115s) [noise=true , address=185.167.204.205:4027 , piece_id=IVPONKBF2PIHZQAW3L4YA4HOMQV5BQML6SKKCJLIEGT5SZSQ2AOA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.438243951 (291h21m26.989714412s) [noise=true , address=185.167.204.205:4022 , piece_id=BES3YK6PYQV2AKOWSAGHF7DGGWGWSWPIJU2ICAQ5WBHKZ5VT5L6A, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.443364331 (291h21m26.994834792s) [noise=true , address=185.167.204.206:4056 , piece_id=EAMZIK6AVJL7IMFF5VOLW3SMCGYKSKB2VXJTNUNLJCVIT7EKXNMQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.445594728 (291h21m26.997065179s) [noise=true , address=185.167.204.205:4016 , piece_id=EHQS3546UXRXTL2MGFCSRTPDDPAYXEYLMISPP5AWY337MHX7HFLQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.449325822 (291h21m27.000796283s) [noise=true , address=185.167.204.205:4031 , piece_id=ZEI2YSIQC655R7WS43WADXGFT5RKT5LFSN6IZEQ6SHTDVOV7BZBQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.449761211 (291h21m27.001231672s) [noise=true , address=185.167.204.205:4018 , piece_id=7OSLPWJLKFTIJPZEAFMZOAH6JAAMX6HKROI25FODFYBPNJWSMLWA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.450419845 (291h21m27.001890306s) [noise=true , address=185.167.204.206:4062 , piece_id=UF3S6NAVGS2KQJPGA5WXP7YG4RQO44P445SNPCWXKXMONZAYUTNQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.453201252 (291h21m27.004671713s) [noise=true , address=185.167.204.205:4007 , piece_id=MEBGOPZ6MFMQY4UZARS2LKJKEMNN5YGSLKPJGTWSKS45CUNXSRGQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:29.661209626 (291h21m28.212680087s) [noise=true , address=89.149.201.147:4016 , piece_id=TGRWGGGBKDANC7YKROWPRMIYNAVFEVE3RXOSUOINFI7KTRS2IYUQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:29.693169182 (291h21m28.244639643s) [noise=true , address=89.149.201.147:4002 , piece_id=UIYJSCMY3LCLRHZUJB6JFJHHXKMYTVKERTZW7F7X44MWJETPY6XA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:29.721204497 (291h21m28.272674958s) [noise=true , address=89.149.201.147:4008 , piece_id=HCX7I4ICPEHDWTYDFIQMKZ5EE5CGPQLYAIHF7WZMTUUU6CNZPHTA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:29.764584711 (291h21m28.316055172s) [noise=true , address=89.149.201.147:4001 , piece_id=QUS6NB3NQBLJFB6YCK3EBDPKCTWEWGKAT3NOR24TVI65SQKHWDOQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:29.768284075 (291h21m28.319754536s) [noise=true , address=89.149.201.147:4011 , piece_id=ZATNMD5HOK35AWGJJ5V66VDJ4YGUVZQJ2OBERQL32ZV5GW2HBSMQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:29.789681961 (291h21m28.341152422s) [noise=true , address=89.149.201.147:4003 , piece_id=RUAYGN2RGF5JJTWFEGIHWH3BHEUDSS5DN2QJ6YLSHPF5BGUXD3RA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:37.298598682 (291h21m35.850069143s) [noise=true , address=81.171.14.70:4043 , piece_id=T3NF6JR3VJIGDMPNUBLMJ3AOTQP5IHOGJYORMCSS7ZMOENQETZ7A, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:37.497478356 (291h21m36.048948818s) [noise=true , address=81.171.14.70:4032 , piece_id=CFH2RTR3BLBUB5ZIZCFC6UHT3FHT3B2XSCHQMKXNGDJJGPHEMCTQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:40.406166933 (291h21m38.957637394s) [noise=true , address=81.171.14.70:4045 , piece_id=BRWQ7G3E547PJE2RLGSESIB3HMP5C4M5NSKAJNGWSFXUBEO6SYEQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:42.511041157 (291h21m41.062511608s) [noise=true , address=81.171.14.70:4029 , piece_id=DKJM5E7EY7LOSBSEBQOZRC6CI5ZPPE47HFOFQJROPCUMUV567CAA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:26:12.012682521 (291h22m10.564152982s) [noise=true , address=81.171.14.70:4042 , piece_id=XNREZEUZ5L5KZ3L7ZOL6IRK7IHU6DJN4RLEQZRFMDKDH2JWC24CQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:26:15.596046283 (291h22m14.147516744s) [noise=true , address=81.171.14.70:4033 , piece_id=JRKPD4EP6FZM5HISOIDJYZY7FAWWQCADAS47FGW2JM5A7RFJR2BQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:26:18.453059659 (291h22m17.00453012s) [noise=true , address=81.171.14.70:4046 , piece_id=OSONRSF6KJS7F2QJVCC3FPZK4AV7QPG5YSM47CW5LIW7EME44AWA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:26:23.442583344 (291h22m21.994053805s) [noise=true , address=81.171.14.70:4034 , piece_id=ITRLKYVNFSYBLABNGNFO3HRJNMX3MWB345FIMWSIPBNAVZPA47UA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:26:23.442621967 (291h22m21.994092428s) [seg_pos={0 0}, upload=256586]: Segment reached optimal threshold of 26 pieces. Nov 21 11:26:23.442836976 (291h22m21.994307437s) [noise=true , address=81.171.14.70:4044 , piece_id=QYUBNGS2FAFHIWMW7GAGYJ3UVIWVP62LIDCOK6QUXEIQNIXRJDAQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1MUVGecysnTSFcJoHUL7QMWb3h6dtXgUd7FbA672fQCyEWhePr): protocol: expected piece hash; context canceled; EOF Nov 21 11:26:23.442845934 (291h22m21.994316395s) [noise=true , address=81.171.14.70:4044 , piece_id=QYUBNGS2FAFHIWMW7GAGYJ3UVIWVP62LIDCOK6QUXEIQNIXRJDAQ, seg_pos={0 0}, upload=256586]: Long tail cancellation due to optimal threshold Nov 21 11:26:23.443227491 (291h22m21.994697942s) [noise=true , address=81.171.14.70:4027 , piece_id=4FDFCCWF3EIROQAFLOHGS743523OYGSXWQSPQEVD3AMQ6WKDQI7Q, seg_pos={0 0}, upload=256586]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122cYvgTxSvC2aEXnBkRk2o8ARTBXj9gqPoa12vW8hogSN6kpA4): protocol: expected piece hash; context canceled; EOF Nov 21 11:26:23.443237770 (291h22m21.994708231s) [noise=true , address=81.171.14.70:4027 , piece_id=4FDFCCWF3EIROQAFLOHGS743523OYGSXWQSPQEVD3AMQ6WKDQI7Q, seg_pos={0 0}, upload=256586]: Long tail cancellation due to optimal threshold Nov 21 11:26:23.443274720 (291h22m21.994745181s) [seg_pos={0 0}, upload=256586]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Nov 21 11:26:23.443378929 (291h22m21.99484939s) [upload=256586]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 21 11:26:23.443387044 (291h22m21.994857495s) [upload=256586]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 21 11:26:23.685139626 (291h22m22.236610077s) [upload=256586]: Done uploading segments... Nov 21 11:26:23.685160676 (291h22m22.236631127s) [upload=256586]: Upload finished. err: <nil> Duration: 5.207209068s core uplink eu-select-1 eu1-placement-15
checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.eu1.linksharing) antiblip: 1 out of 60 is failed (threshold=6) [..X.........................................................] duration: 257.270823ms remote: "tcp:207.211.208.129:443" target: https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt trace: https://jaeger.dev.storj.io/trace/e2bae25cf885024 FAIL (2025-11-21 11:46:22 UTC) orbiter warning: Get "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt": context deadline exceeded Duration: 257.299056ms edge linksharing bgp eu1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 2 out of 60 is failed (threshold=3) [...........................X.X..............................] duration: 5.989277089s trace: https://jaeger.dev.storj.io/trace/a7f69ee106efb7f FAIL (2025-11-21 11:46:09 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== upload=256586]: Uploading piece... Nov 21 11:25:27.210346024 (291h21m25.761816485s) [noise=true , address=89.149.201.147:4008 , piece_id=HCX7I4ICPEHDWTYDFIQMKZ5EE5CGPQLYAIHF7WZMTUUU6CNZPHTA, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.211425289 (291h21m25.76289575s) [noise=true , address=81.171.14.70:4044 , piece_id=QYUBNGS2FAFHIWMW7GAGYJ3UVIWVP62LIDCOK6QUXEIQNIXRJDAQ, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.213103795 (291h21m25.764574246s) [noise=true , address=185.167.204.205:4022 , piece_id=BES3YK6PYQV2AKOWSAGHF7DGGWGWSWPIJU2ICAQ5WBHKZ5VT5L6A, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.213408395 (291h21m25.764878856s) [noise=true , address=185.167.204.205:4027 , piece_id=IVPONKBF2PIHZQAW3L4YA4HOMQV5BQML6SKKCJLIEGT5SZSQ2AOA, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.214501747 (291h21m25.765972208s) [noise=true , address=81.171.14.70:4042 , piece_id=XNREZEUZ5L5KZ3L7ZOL6IRK7IHU6DJN4RLEQZRFMDKDH2JWC24CQ, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.215013752 (291h21m25.766484213s) [noise=true , address=81.171.14.70:4034 , piece_id=ITRLKYVNFSYBLABNGNFO3HRJNMX3MWB345FIMWSIPBNAVZPA47UA, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.215087572 (291h21m25.766558044s) [noise=true , address=185.167.204.205:4031 , piece_id=ZEI2YSIQC655R7WS43WADXGFT5RKT5LFSN6IZEQ6SHTDVOV7BZBQ, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.216432844 (291h21m25.767903305s) [noise=true , address=185.167.204.206:4056 , piece_id=EAMZIK6AVJL7IMFF5VOLW3SMCGYKSKB2VXJTNUNLJCVIT7EKXNMQ, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.216662281 (291h21m25.768132742s) [noise=true , address=89.149.201.147:4011 , piece_id=ZATNMD5HOK35AWGJJ5V66VDJ4YGUVZQJ2OBERQL32ZV5GW2HBSMQ, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.218008314 (291h21m25.769478775s) [noise=true , address=81.171.14.70:4032 , piece_id=CFH2RTR3BLBUB5ZIZCFC6UHT3FHT3B2XSCHQMKXNGDJJGPHEMCTQ, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:28.035150976 (291h21m26.586621437s) [upload=256586]: Next returned nil segment Nov 21 11:25:28.035169622 (291h21m26.586640083s) [upload=256586]: Waiting for error group managing segments... Nov 21 11:25:28.416646836 (291h21m26.968117317s) [noise=true , address=185.167.204.206:4064 , piece_id=SRFIWFQLRAGCNIKYLHMGT4QRUL3WH2QPLLIKAJ7DLFJEQUW5DPPA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.417338854 (291h21m26.968809325s) [noise=true , address=185.167.204.205:4033 , piece_id=FEWQVHPIKN3Q6O5AZ6AGJBLUPZ34PBZFB5X3XNSKGPOGI4AIO7JA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.421089406 (291h21m26.972559867s) [noise=true , address=185.167.204.206:4072 , piece_id=GYCO53OWGOMOHK4FHC5GENN4LTQGSZKZKOI5V534RLV3LHLPKC6Q, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.426252887 (291h21m26.977723348s) [noise=true , address=185.167.204.206:4046 , piece_id=BZ57ZA35ITEOA6CAZJGNODK5MUTXEZWDV7E2OGBZOSMS4OQZYAEA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.431720689 (291h21m26.98319115s) [noise=true , address=185.167.204.205:4027 , piece_id=IVPONKBF2PIHZQAW3L4YA4HOMQV5BQML6SKKCJLIEGT5SZSQ2AOA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.438243951 (291h21m26.989714412s) [noise=true , address=185.167.204.205:4022 , piece_id=BES3YK6PYQV2AKOWSAGHF7DGGWGWSWPIJU2ICAQ5WBHKZ5VT5L6A, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.443364331 (291h21m26.994834792s) [noise=true , address=185.167.204.206:4056 , piece_id=EAMZIK6AVJL7IMFF5VOLW3SMCGYKSKB2VXJTNUNLJCVIT7EKXNMQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.445594728 (291h21m26.997065179s) [noise=true , address=185.167.204.205:4016 , piece_id=EHQS3546UXRXTL2MGFCSRTPDDPAYXEYLMISPP5AWY337MHX7HFLQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.449325822 (291h21m27.000796283s) [noise=true , address=185.167.204.205:4031 , piece_id=ZEI2YSIQC655R7WS43WADXGFT5RKT5LFSN6IZEQ6SHTDVOV7BZBQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.449761211 (291h21m27.001231672s) [noise=true , address=185.167.204.205:4018 , piece_id=7OSLPWJLKFTIJPZEAFMZOAH6JAAMX6HKROI25FODFYBPNJWSMLWA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.450419845 (291h21m27.001890306s) [noise=true , address=185.167.204.206:4062 , piece_id=UF3S6NAVGS2KQJPGA5WXP7YG4RQO44P445SNPCWXKXMONZAYUTNQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.453201252 (291h21m27.004671713s) [noise=true , address=185.167.204.205:4007 , piece_id=MEBGOPZ6MFMQY4UZARS2LKJKEMNN5YGSLKPJGTWSKS45CUNXSRGQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:29.661209626 (291h21m28.212680087s) [noise=true , address=89.149.201.147:4016 , piece_id=TGRWGGGBKDANC7YKROWPRMIYNAVFEVE3RXOSUOINFI7KTRS2IYUQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:29.693169182 (291h21m28.244639643s) [noise=true , address=89.149.201.147:4002 , piece_id=UIYJSCMY3LCLRHZUJB6JFJHHXKMYTVKERTZW7F7X44MWJETPY6XA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:29.721204497 (291h21m28.272674958s) [noise=true , address=89.149.201.147:4008 , piece_id=HCX7I4ICPEHDWTYDFIQMKZ5EE5CGPQLYAIHF7WZMTUUU6CNZPHTA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:29.764584711 (291h21m28.316055172s) [noise=true , address=89.149.201.147:4001 , piece_id=QUS6NB3NQBLJFB6YCK3EBDPKCTWEWGKAT3NOR24TVI65SQKHWDOQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:29.768284075 (291h21m28.319754536s) [noise=true , address=89.149.201.147:4011 , piece_id=ZATNMD5HOK35AWGJJ5V66VDJ4YGUVZQJ2OBERQL32ZV5GW2HBSMQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:29.789681961 (291h21m28.341152422s) [noise=true , address=89.149.201.147:4003 , piece_id=RUAYGN2RGF5JJTWFEGIHWH3BHEUDSS5DN2QJ6YLSHPF5BGUXD3RA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:37.298598682 (291h21m35.850069143s) [noise=true , address=81.171.14.70:4043 , piece_id=T3NF6JR3VJIGDMPNUBLMJ3AOTQP5IHOGJYORMCSS7ZMOENQETZ7A, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:37.497478356 (291h21m36.048948818s) [noise=true , address=81.171.14.70:4032 , piece_id=CFH2RTR3BLBUB5ZIZCFC6UHT3FHT3B2XSCHQMKXNGDJJGPHEMCTQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:40.406166933 (291h21m38.957637394s) [noise=true , address=81.171.14.70:4045 , piece_id=BRWQ7G3E547PJE2RLGSESIB3HMP5C4M5NSKAJNGWSFXUBEO6SYEQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:42.511041157 (291h21m41.062511608s) [noise=true , address=81.171.14.70:4029 , piece_id=DKJM5E7EY7LOSBSEBQOZRC6CI5ZPPE47HFOFQJROPCUMUV567CAA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:26:12.012682521 (291h22m10.564152982s) [noise=true , address=81.171.14.70:4042 , piece_id=XNREZEUZ5L5KZ3L7ZOL6IRK7IHU6DJN4RLEQZRFMDKDH2JWC24CQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:26:15.596046283 (291h22m14.147516744s) [noise=true , address=81.171.14.70:4033 , piece_id=JRKPD4EP6FZM5HISOIDJYZY7FAWWQCADAS47FGW2JM5A7RFJR2BQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:26:18.453059659 (291h22m17.00453012s) [noise=true , address=81.171.14.70:4046 , piece_id=OSONRSF6KJS7F2QJVCC3FPZK4AV7QPG5YSM47CW5LIW7EME44AWA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:26:23.442583344 (291h22m21.994053805s) [noise=true , address=81.171.14.70:4034 , piece_id=ITRLKYVNFSYBLABNGNFO3HRJNMX3MWB345FIMWSIPBNAVZPA47UA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:26:23.442621967 (291h22m21.994092428s) [seg_pos={0 0}, upload=256586]: Segment reached optimal threshold of 26 pieces. Nov 21 11:26:23.442836976 (291h22m21.994307437s) [noise=true , address=81.171.14.70:4044 , piece_id=QYUBNGS2FAFHIWMW7GAGYJ3UVIWVP62LIDCOK6QUXEIQNIXRJDAQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1MUVGecysnTSFcJoHUL7QMWb3h6dtXgUd7FbA672fQCyEWhePr): protocol: expected piece hash; context canceled; EOF Nov 21 11:26:23.442845934 (291h22m21.994316395s) [noise=true , address=81.171.14.70:4044 , piece_id=QYUBNGS2FAFHIWMW7GAGYJ3UVIWVP62LIDCOK6QUXEIQNIXRJDAQ, seg_pos={0 0}, upload=256586]: Long tail cancellation due to optimal threshold Nov 21 11:26:23.443227491 (291h22m21.994697942s) [noise=true , address=81.171.14.70:4027 , piece_id=4FDFCCWF3EIROQAFLOHGS743523OYGSXWQSPQEVD3AMQ6WKDQI7Q, seg_pos={0 0}, upload=256586]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122cYvgTxSvC2aEXnBkRk2o8ARTBXj9gqPoa12vW8hogSN6kpA4): protocol: expected piece hash; context canceled; EOF Nov 21 11:26:23.443237770 (291h22m21.994708231s) [noise=true , address=81.171.14.70:4027 , piece_id=4FDFCCWF3EIROQAFLOHGS743523OYGSXWQSPQEVD3AMQ6WKDQI7Q, seg_pos={0 0}, upload=256586]: Long tail cancellation due to optimal threshold Nov 21 11:26:23.443274720 (291h22m21.994745181s) [seg_pos={0 0}, upload=256586]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Nov 21 11:26:23.443378929 (291h22m21.99484939s) [upload=256586]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 21 11:26:23.443387044 (291h22m21.994857495s) [upload=256586]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 21 11:26:23.685139626 (291h22m22.236610077s) [upload=256586]: Done uploading segments... Nov 21 11:26:23.685160676 (291h22m22.236631127s) [upload=256586]: Upload finished. err: <nil> Duration: 5.989311054s core uplink eu-select-1 eu1-placement-15
checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.eu1.linksharing) antiblip: 1 out of 60 is failed (threshold=6) [...X........................................................] duration: 254.022587ms remote: "tcp:207.211.208.129:443" target: https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt trace: https://jaeger.dev.storj.io/trace/e2bae25cf885024 FAIL (2025-11-21 11:46:04 UTC) orbiter warning: Get "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt": context deadline exceeded Duration: 254.094794ms edge linksharing bgp eu1
checkers.HTTP{"GET" "https://link.storjshare.io/raw/jw7f74fzbtwblz3mwqolmnivikxq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.us1.linksharing) antiblip: 1 out of 60 is failed (threshold=6) [.X..........................................................] duration: 255.965727ms remote: "tcp:207.211.208.129:443" target: https://link.storjshare.io/raw/jw7f74fzbtwblz3mwqolmnivikxq/uptime-checks/hello-world.txt trace: https://jaeger.dev.storj.io/trace/41cfd75a627799ce FAIL (2025-11-21 11:45:59 UTC) orbiter warning: Get "https://link.storjshare.io/raw/jw7f74fzbtwblz3mwqolmnivikxq/uptime-checks/hello-world.txt": context deadline exceeded Duration: 255.993559ms edge linksharing bgp us1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 2 out of 60 is failed (threshold=3) [............................X.X.............................] duration: 13.710142719s trace: https://jaeger.dev.storj.io/trace/a7f69ee106efb7f FAIL (2025-11-21 11:45:52 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== upload=256586]: Uploading piece... Nov 21 11:25:27.210346024 (291h21m25.761816485s) [noise=true , address=89.149.201.147:4008 , piece_id=HCX7I4ICPEHDWTYDFIQMKZ5EE5CGPQLYAIHF7WZMTUUU6CNZPHTA, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.211425289 (291h21m25.76289575s) [noise=true , address=81.171.14.70:4044 , piece_id=QYUBNGS2FAFHIWMW7GAGYJ3UVIWVP62LIDCOK6QUXEIQNIXRJDAQ, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.213103795 (291h21m25.764574246s) [noise=true , address=185.167.204.205:4022 , piece_id=BES3YK6PYQV2AKOWSAGHF7DGGWGWSWPIJU2ICAQ5WBHKZ5VT5L6A, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.213408395 (291h21m25.764878856s) [noise=true , address=185.167.204.205:4027 , piece_id=IVPONKBF2PIHZQAW3L4YA4HOMQV5BQML6SKKCJLIEGT5SZSQ2AOA, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.214501747 (291h21m25.765972208s) [noise=true , address=81.171.14.70:4042 , piece_id=XNREZEUZ5L5KZ3L7ZOL6IRK7IHU6DJN4RLEQZRFMDKDH2JWC24CQ, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.215013752 (291h21m25.766484213s) [noise=true , address=81.171.14.70:4034 , piece_id=ITRLKYVNFSYBLABNGNFO3HRJNMX3MWB345FIMWSIPBNAVZPA47UA, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.215087572 (291h21m25.766558044s) [noise=true , address=185.167.204.205:4031 , piece_id=ZEI2YSIQC655R7WS43WADXGFT5RKT5LFSN6IZEQ6SHTDVOV7BZBQ, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.216432844 (291h21m25.767903305s) [noise=true , address=185.167.204.206:4056 , piece_id=EAMZIK6AVJL7IMFF5VOLW3SMCGYKSKB2VXJTNUNLJCVIT7EKXNMQ, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.216662281 (291h21m25.768132742s) [noise=true , address=89.149.201.147:4011 , piece_id=ZATNMD5HOK35AWGJJ5V66VDJ4YGUVZQJ2OBERQL32ZV5GW2HBSMQ, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.218008314 (291h21m25.769478775s) [noise=true , address=81.171.14.70:4032 , piece_id=CFH2RTR3BLBUB5ZIZCFC6UHT3FHT3B2XSCHQMKXNGDJJGPHEMCTQ, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:28.035150976 (291h21m26.586621437s) [upload=256586]: Next returned nil segment Nov 21 11:25:28.035169622 (291h21m26.586640083s) [upload=256586]: Waiting for error group managing segments... Nov 21 11:25:28.416646836 (291h21m26.968117317s) [noise=true , address=185.167.204.206:4064 , piece_id=SRFIWFQLRAGCNIKYLHMGT4QRUL3WH2QPLLIKAJ7DLFJEQUW5DPPA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.417338854 (291h21m26.968809325s) [noise=true , address=185.167.204.205:4033 , piece_id=FEWQVHPIKN3Q6O5AZ6AGJBLUPZ34PBZFB5X3XNSKGPOGI4AIO7JA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.421089406 (291h21m26.972559867s) [noise=true , address=185.167.204.206:4072 , piece_id=GYCO53OWGOMOHK4FHC5GENN4LTQGSZKZKOI5V534RLV3LHLPKC6Q, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.426252887 (291h21m26.977723348s) [noise=true , address=185.167.204.206:4046 , piece_id=BZ57ZA35ITEOA6CAZJGNODK5MUTXEZWDV7E2OGBZOSMS4OQZYAEA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.431720689 (291h21m26.98319115s) [noise=true , address=185.167.204.205:4027 , piece_id=IVPONKBF2PIHZQAW3L4YA4HOMQV5BQML6SKKCJLIEGT5SZSQ2AOA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.438243951 (291h21m26.989714412s) [noise=true , address=185.167.204.205:4022 , piece_id=BES3YK6PYQV2AKOWSAGHF7DGGWGWSWPIJU2ICAQ5WBHKZ5VT5L6A, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.443364331 (291h21m26.994834792s) [noise=true , address=185.167.204.206:4056 , piece_id=EAMZIK6AVJL7IMFF5VOLW3SMCGYKSKB2VXJTNUNLJCVIT7EKXNMQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.445594728 (291h21m26.997065179s) [noise=true , address=185.167.204.205:4016 , piece_id=EHQS3546UXRXTL2MGFCSRTPDDPAYXEYLMISPP5AWY337MHX7HFLQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.449325822 (291h21m27.000796283s) [noise=true , address=185.167.204.205:4031 , piece_id=ZEI2YSIQC655R7WS43WADXGFT5RKT5LFSN6IZEQ6SHTDVOV7BZBQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.449761211 (291h21m27.001231672s) [noise=true , address=185.167.204.205:4018 , piece_id=7OSLPWJLKFTIJPZEAFMZOAH6JAAMX6HKROI25FODFYBPNJWSMLWA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.450419845 (291h21m27.001890306s) [noise=true , address=185.167.204.206:4062 , piece_id=UF3S6NAVGS2KQJPGA5WXP7YG4RQO44P445SNPCWXKXMONZAYUTNQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.453201252 (291h21m27.004671713s) [noise=true , address=185.167.204.205:4007 , piece_id=MEBGOPZ6MFMQY4UZARS2LKJKEMNN5YGSLKPJGTWSKS45CUNXSRGQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:29.661209626 (291h21m28.212680087s) [noise=true , address=89.149.201.147:4016 , piece_id=TGRWGGGBKDANC7YKROWPRMIYNAVFEVE3RXOSUOINFI7KTRS2IYUQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:29.693169182 (291h21m28.244639643s) [noise=true , address=89.149.201.147:4002 , piece_id=UIYJSCMY3LCLRHZUJB6JFJHHXKMYTVKERTZW7F7X44MWJETPY6XA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:29.721204497 (291h21m28.272674958s) [noise=true , address=89.149.201.147:4008 , piece_id=HCX7I4ICPEHDWTYDFIQMKZ5EE5CGPQLYAIHF7WZMTUUU6CNZPHTA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:29.764584711 (291h21m28.316055172s) [noise=true , address=89.149.201.147:4001 , piece_id=QUS6NB3NQBLJFB6YCK3EBDPKCTWEWGKAT3NOR24TVI65SQKHWDOQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:29.768284075 (291h21m28.319754536s) [noise=true , address=89.149.201.147:4011 , piece_id=ZATNMD5HOK35AWGJJ5V66VDJ4YGUVZQJ2OBERQL32ZV5GW2HBSMQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:29.789681961 (291h21m28.341152422s) [noise=true , address=89.149.201.147:4003 , piece_id=RUAYGN2RGF5JJTWFEGIHWH3BHEUDSS5DN2QJ6YLSHPF5BGUXD3RA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:37.298598682 (291h21m35.850069143s) [noise=true , address=81.171.14.70:4043 , piece_id=T3NF6JR3VJIGDMPNUBLMJ3AOTQP5IHOGJYORMCSS7ZMOENQETZ7A, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:37.497478356 (291h21m36.048948818s) [noise=true , address=81.171.14.70:4032 , piece_id=CFH2RTR3BLBUB5ZIZCFC6UHT3FHT3B2XSCHQMKXNGDJJGPHEMCTQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:40.406166933 (291h21m38.957637394s) [noise=true , address=81.171.14.70:4045 , piece_id=BRWQ7G3E547PJE2RLGSESIB3HMP5C4M5NSKAJNGWSFXUBEO6SYEQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:42.511041157 (291h21m41.062511608s) [noise=true , address=81.171.14.70:4029 , piece_id=DKJM5E7EY7LOSBSEBQOZRC6CI5ZPPE47HFOFQJROPCUMUV567CAA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:26:12.012682521 (291h22m10.564152982s) [noise=true , address=81.171.14.70:4042 , piece_id=XNREZEUZ5L5KZ3L7ZOL6IRK7IHU6DJN4RLEQZRFMDKDH2JWC24CQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:26:15.596046283 (291h22m14.147516744s) [noise=true , address=81.171.14.70:4033 , piece_id=JRKPD4EP6FZM5HISOIDJYZY7FAWWQCADAS47FGW2JM5A7RFJR2BQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:26:18.453059659 (291h22m17.00453012s) [noise=true , address=81.171.14.70:4046 , piece_id=OSONRSF6KJS7F2QJVCC3FPZK4AV7QPG5YSM47CW5LIW7EME44AWA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:26:23.442583344 (291h22m21.994053805s) [noise=true , address=81.171.14.70:4034 , piece_id=ITRLKYVNFSYBLABNGNFO3HRJNMX3MWB345FIMWSIPBNAVZPA47UA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:26:23.442621967 (291h22m21.994092428s) [seg_pos={0 0}, upload=256586]: Segment reached optimal threshold of 26 pieces. Nov 21 11:26:23.442836976 (291h22m21.994307437s) [noise=true , address=81.171.14.70:4044 , piece_id=QYUBNGS2FAFHIWMW7GAGYJ3UVIWVP62LIDCOK6QUXEIQNIXRJDAQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1MUVGecysnTSFcJoHUL7QMWb3h6dtXgUd7FbA672fQCyEWhePr): protocol: expected piece hash; context canceled; EOF Nov 21 11:26:23.442845934 (291h22m21.994316395s) [noise=true , address=81.171.14.70:4044 , piece_id=QYUBNGS2FAFHIWMW7GAGYJ3UVIWVP62LIDCOK6QUXEIQNIXRJDAQ, seg_pos={0 0}, upload=256586]: Long tail cancellation due to optimal threshold Nov 21 11:26:23.443227491 (291h22m21.994697942s) [noise=true , address=81.171.14.70:4027 , piece_id=4FDFCCWF3EIROQAFLOHGS743523OYGSXWQSPQEVD3AMQ6WKDQI7Q, seg_pos={0 0}, upload=256586]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122cYvgTxSvC2aEXnBkRk2o8ARTBXj9gqPoa12vW8hogSN6kpA4): protocol: expected piece hash; context canceled; EOF Nov 21 11:26:23.443237770 (291h22m21.994708231s) [noise=true , address=81.171.14.70:4027 , piece_id=4FDFCCWF3EIROQAFLOHGS743523OYGSXWQSPQEVD3AMQ6WKDQI7Q, seg_pos={0 0}, upload=256586]: Long tail cancellation due to optimal threshold Nov 21 11:26:23.443274720 (291h22m21.994745181s) [seg_pos={0 0}, upload=256586]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Nov 21 11:26:23.443378929 (291h22m21.99484939s) [upload=256586]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 21 11:26:23.443387044 (291h22m21.994857495s) [upload=256586]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 21 11:26:23.685139626 (291h22m22.236610077s) [upload=256586]: Done uploading segments... Nov 21 11:26:23.685160676 (291h22m22.236631127s) [upload=256586]: Upload finished. err: <nil> Duration: 13.710175139s core uplink eu-select-1 eu1-placement-15
checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.eu1.linksharing) antiblip: 1 out of 60 is failed (threshold=6) [....X.......................................................] duration: 256.781351ms remote: "tcp:207.211.208.129:443" target: https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt trace: https://jaeger.dev.storj.io/trace/e2bae25cf885024 FAIL (2025-11-21 11:45:44 UTC) orbiter warning: Get "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt": context deadline exceeded Duration: 256.815395ms edge linksharing bgp eu1
checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jwhnbh22a7huxgm5mpnu45kaahuq&Signature=gznnRNzM%2FpTrDIdAeC%2Bpz3LKg0A%3D&Expires=1771549318", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.eu1.gateway) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 247.978298ms remote: "tcp:207.211.208.129:443" target: https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jwhnbh22a7huxgm5mpnu45kaahuq&Signature=gznnRNzM%2FpTrDIdAeC%2Bpz3LKg0A%3D&Expires=1771549318 trace: https://jaeger.dev.storj.io/trace/5285054b3a776da7 FAIL (2025-11-21 11:45:43 UTC) orbiter warning: Get "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jwhnbh22a7huxgm5mpnu45kaahuq&Signature=gznnRNzM%2FpTrDIdAeC%2Bpz3LKg0A%3D&Expires=1771549318": context deadline exceeded Duration: 248.006482ms edge gateway vhost bgp eu1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 49 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX...........] duration: 3.404045928s trace: https://jaeger.dev.storj.io/trace/6af15a3a67b655d5 FAIL (2025-11-21 11:45:39 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 3.404071316s core uplink us-select-1 us1-placement-20
checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us1.gateway) antiblip: 1 out of 60 is failed (threshold=3) [.....X......................................................] duration: 257.797807ms remote: "tcp:207.211.208.129:443" target: https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222 trace: https://jaeger.dev.storj.io/trace/3e5c193367f9a974 FAIL (2025-11-21 11:45:29 UTC) orbiter warning: Get "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222": context deadline exceeded Duration: 257.823736ms edge gateway vhost bgp us1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 2 out of 60 is failed (threshold=3) [.............................X.X............................] duration: 7.210163631s trace: https://jaeger.dev.storj.io/trace/a7f69ee106efb7f FAIL (2025-11-21 11:45:07 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== upload=256586]: Uploading piece... Nov 21 11:25:27.210346024 (291h21m25.761816485s) [noise=true , address=89.149.201.147:4008 , piece_id=HCX7I4ICPEHDWTYDFIQMKZ5EE5CGPQLYAIHF7WZMTUUU6CNZPHTA, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.211425289 (291h21m25.76289575s) [noise=true , address=81.171.14.70:4044 , piece_id=QYUBNGS2FAFHIWMW7GAGYJ3UVIWVP62LIDCOK6QUXEIQNIXRJDAQ, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.213103795 (291h21m25.764574246s) [noise=true , address=185.167.204.205:4022 , piece_id=BES3YK6PYQV2AKOWSAGHF7DGGWGWSWPIJU2ICAQ5WBHKZ5VT5L6A, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.213408395 (291h21m25.764878856s) [noise=true , address=185.167.204.205:4027 , piece_id=IVPONKBF2PIHZQAW3L4YA4HOMQV5BQML6SKKCJLIEGT5SZSQ2AOA, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.214501747 (291h21m25.765972208s) [noise=true , address=81.171.14.70:4042 , piece_id=XNREZEUZ5L5KZ3L7ZOL6IRK7IHU6DJN4RLEQZRFMDKDH2JWC24CQ, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.215013752 (291h21m25.766484213s) [noise=true , address=81.171.14.70:4034 , piece_id=ITRLKYVNFSYBLABNGNFO3HRJNMX3MWB345FIMWSIPBNAVZPA47UA, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.215087572 (291h21m25.766558044s) [noise=true , address=185.167.204.205:4031 , piece_id=ZEI2YSIQC655R7WS43WADXGFT5RKT5LFSN6IZEQ6SHTDVOV7BZBQ, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.216432844 (291h21m25.767903305s) [noise=true , address=185.167.204.206:4056 , piece_id=EAMZIK6AVJL7IMFF5VOLW3SMCGYKSKB2VXJTNUNLJCVIT7EKXNMQ, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.216662281 (291h21m25.768132742s) [noise=true , address=89.149.201.147:4011 , piece_id=ZATNMD5HOK35AWGJJ5V66VDJ4YGUVZQJ2OBERQL32ZV5GW2HBSMQ, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.218008314 (291h21m25.769478775s) [noise=true , address=81.171.14.70:4032 , piece_id=CFH2RTR3BLBUB5ZIZCFC6UHT3FHT3B2XSCHQMKXNGDJJGPHEMCTQ, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:28.035150976 (291h21m26.586621437s) [upload=256586]: Next returned nil segment Nov 21 11:25:28.035169622 (291h21m26.586640083s) [upload=256586]: Waiting for error group managing segments... Nov 21 11:25:28.416646836 (291h21m26.968117317s) [noise=true , address=185.167.204.206:4064 , piece_id=SRFIWFQLRAGCNIKYLHMGT4QRUL3WH2QPLLIKAJ7DLFJEQUW5DPPA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.417338854 (291h21m26.968809325s) [noise=true , address=185.167.204.205:4033 , piece_id=FEWQVHPIKN3Q6O5AZ6AGJBLUPZ34PBZFB5X3XNSKGPOGI4AIO7JA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.421089406 (291h21m26.972559867s) [noise=true , address=185.167.204.206:4072 , piece_id=GYCO53OWGOMOHK4FHC5GENN4LTQGSZKZKOI5V534RLV3LHLPKC6Q, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.426252887 (291h21m26.977723348s) [noise=true , address=185.167.204.206:4046 , piece_id=BZ57ZA35ITEOA6CAZJGNODK5MUTXEZWDV7E2OGBZOSMS4OQZYAEA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.431720689 (291h21m26.98319115s) [noise=true , address=185.167.204.205:4027 , piece_id=IVPONKBF2PIHZQAW3L4YA4HOMQV5BQML6SKKCJLIEGT5SZSQ2AOA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.438243951 (291h21m26.989714412s) [noise=true , address=185.167.204.205:4022 , piece_id=BES3YK6PYQV2AKOWSAGHF7DGGWGWSWPIJU2ICAQ5WBHKZ5VT5L6A, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.443364331 (291h21m26.994834792s) [noise=true , address=185.167.204.206:4056 , piece_id=EAMZIK6AVJL7IMFF5VOLW3SMCGYKSKB2VXJTNUNLJCVIT7EKXNMQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.445594728 (291h21m26.997065179s) [noise=true , address=185.167.204.205:4016 , piece_id=EHQS3546UXRXTL2MGFCSRTPDDPAYXEYLMISPP5AWY337MHX7HFLQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.449325822 (291h21m27.000796283s) [noise=true , address=185.167.204.205:4031 , piece_id=ZEI2YSIQC655R7WS43WADXGFT5RKT5LFSN6IZEQ6SHTDVOV7BZBQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.449761211 (291h21m27.001231672s) [noise=true , address=185.167.204.205:4018 , piece_id=7OSLPWJLKFTIJPZEAFMZOAH6JAAMX6HKROI25FODFYBPNJWSMLWA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.450419845 (291h21m27.001890306s) [noise=true , address=185.167.204.206:4062 , piece_id=UF3S6NAVGS2KQJPGA5WXP7YG4RQO44P445SNPCWXKXMONZAYUTNQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.453201252 (291h21m27.004671713s) [noise=true , address=185.167.204.205:4007 , piece_id=MEBGOPZ6MFMQY4UZARS2LKJKEMNN5YGSLKPJGTWSKS45CUNXSRGQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:29.661209626 (291h21m28.212680087s) [noise=true , address=89.149.201.147:4016 , piece_id=TGRWGGGBKDANC7YKROWPRMIYNAVFEVE3RXOSUOINFI7KTRS2IYUQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:29.693169182 (291h21m28.244639643s) [noise=true , address=89.149.201.147:4002 , piece_id=UIYJSCMY3LCLRHZUJB6JFJHHXKMYTVKERTZW7F7X44MWJETPY6XA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:29.721204497 (291h21m28.272674958s) [noise=true , address=89.149.201.147:4008 , piece_id=HCX7I4ICPEHDWTYDFIQMKZ5EE5CGPQLYAIHF7WZMTUUU6CNZPHTA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:29.764584711 (291h21m28.316055172s) [noise=true , address=89.149.201.147:4001 , piece_id=QUS6NB3NQBLJFB6YCK3EBDPKCTWEWGKAT3NOR24TVI65SQKHWDOQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:29.768284075 (291h21m28.319754536s) [noise=true , address=89.149.201.147:4011 , piece_id=ZATNMD5HOK35AWGJJ5V66VDJ4YGUVZQJ2OBERQL32ZV5GW2HBSMQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:29.789681961 (291h21m28.341152422s) [noise=true , address=89.149.201.147:4003 , piece_id=RUAYGN2RGF5JJTWFEGIHWH3BHEUDSS5DN2QJ6YLSHPF5BGUXD3RA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:37.298598682 (291h21m35.850069143s) [noise=true , address=81.171.14.70:4043 , piece_id=T3NF6JR3VJIGDMPNUBLMJ3AOTQP5IHOGJYORMCSS7ZMOENQETZ7A, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:37.497478356 (291h21m36.048948818s) [noise=true , address=81.171.14.70:4032 , piece_id=CFH2RTR3BLBUB5ZIZCFC6UHT3FHT3B2XSCHQMKXNGDJJGPHEMCTQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:40.406166933 (291h21m38.957637394s) [noise=true , address=81.171.14.70:4045 , piece_id=BRWQ7G3E547PJE2RLGSESIB3HMP5C4M5NSKAJNGWSFXUBEO6SYEQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:42.511041157 (291h21m41.062511608s) [noise=true , address=81.171.14.70:4029 , piece_id=DKJM5E7EY7LOSBSEBQOZRC6CI5ZPPE47HFOFQJROPCUMUV567CAA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:26:12.012682521 (291h22m10.564152982s) [noise=true , address=81.171.14.70:4042 , piece_id=XNREZEUZ5L5KZ3L7ZOL6IRK7IHU6DJN4RLEQZRFMDKDH2JWC24CQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:26:15.596046283 (291h22m14.147516744s) [noise=true , address=81.171.14.70:4033 , piece_id=JRKPD4EP6FZM5HISOIDJYZY7FAWWQCADAS47FGW2JM5A7RFJR2BQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:26:18.453059659 (291h22m17.00453012s) [noise=true , address=81.171.14.70:4046 , piece_id=OSONRSF6KJS7F2QJVCC3FPZK4AV7QPG5YSM47CW5LIW7EME44AWA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:26:23.442583344 (291h22m21.994053805s) [noise=true , address=81.171.14.70:4034 , piece_id=ITRLKYVNFSYBLABNGNFO3HRJNMX3MWB345FIMWSIPBNAVZPA47UA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:26:23.442621967 (291h22m21.994092428s) [seg_pos={0 0}, upload=256586]: Segment reached optimal threshold of 26 pieces. Nov 21 11:26:23.442836976 (291h22m21.994307437s) [noise=true , address=81.171.14.70:4044 , piece_id=QYUBNGS2FAFHIWMW7GAGYJ3UVIWVP62LIDCOK6QUXEIQNIXRJDAQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1MUVGecysnTSFcJoHUL7QMWb3h6dtXgUd7FbA672fQCyEWhePr): protocol: expected piece hash; context canceled; EOF Nov 21 11:26:23.442845934 (291h22m21.994316395s) [noise=true , address=81.171.14.70:4044 , piece_id=QYUBNGS2FAFHIWMW7GAGYJ3UVIWVP62LIDCOK6QUXEIQNIXRJDAQ, seg_pos={0 0}, upload=256586]: Long tail cancellation due to optimal threshold Nov 21 11:26:23.443227491 (291h22m21.994697942s) [noise=true , address=81.171.14.70:4027 , piece_id=4FDFCCWF3EIROQAFLOHGS743523OYGSXWQSPQEVD3AMQ6WKDQI7Q, seg_pos={0 0}, upload=256586]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122cYvgTxSvC2aEXnBkRk2o8ARTBXj9gqPoa12vW8hogSN6kpA4): protocol: expected piece hash; context canceled; EOF Nov 21 11:26:23.443237770 (291h22m21.994708231s) [noise=true , address=81.171.14.70:4027 , piece_id=4FDFCCWF3EIROQAFLOHGS743523OYGSXWQSPQEVD3AMQ6WKDQI7Q, seg_pos={0 0}, upload=256586]: Long tail cancellation due to optimal threshold Nov 21 11:26:23.443274720 (291h22m21.994745181s) [seg_pos={0 0}, upload=256586]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Nov 21 11:26:23.443378929 (291h22m21.99484939s) [upload=256586]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 21 11:26:23.443387044 (291h22m21.994857495s) [upload=256586]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 21 11:26:23.685139626 (291h22m22.236610077s) [upload=256586]: Done uploading segments... Nov 21 11:26:23.685160676 (291h22m22.236631127s) [upload=256586]: Upload finished. err: <nil> Duration: 7.210202133s core uplink eu-select-1 eu1-placement-15

Orbiter Check History

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us1.gateway)

checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us1.gateway)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.eu1.linksharing)

checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us1.gateway)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.eu1.linksharing)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.HTTP{"GET" "https://link.storjshare.io/raw/jw7f74fzbtwblz3mwqolmnivikxq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.us1.linksharing)

checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us1.gateway)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us1.gateway)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.eu1.linksharing)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.eu1.linksharing)

checkers.HTTP{"GET" "https://link.storjshare.io/raw/jw7f74fzbtwblz3mwqolmnivikxq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.us1.linksharing)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.eu1.linksharing)

checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jwhnbh22a7huxgm5mpnu45kaahuq&Signature=gznnRNzM%2FpTrDIdAeC%2Bpz3LKg0A%3D&Expires=1771549318", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.eu1.gateway)

checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us1.gateway)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)