Orbiter: EU/US/AP | Code | Config | Playbook | Edge Deployment | Escalation Procedure
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
14 out of 60 is failed (threshold=3) [.X......X....X...X..X.X.X.....X.......X.X.....X....XX...X...] duration:
30.002336918s trace:
https://jaeger.dev.storj.io/trace/25efa104cb3576c8
FAIL (2025-12-30 06:50:14 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW, Piece ID: DG5N5IK35GLLLBJKWK7J53N3ER737OAGQBCHQ53AVMTIJHNWRXEA) context deadline exceeded; piecestore: (Node ID: 1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w, Piece ID: TSCFP4MGODAU4IOWHIKWBKNNAE4G2ZGCBTVRKF3KVXCCP7Z6H7EQ) context deadline exceeded
======LOGS======
30 06:47:33.902700925 (591h18m33.77378396s) [upload=518373]: Starting upload
Dec 30 06:47:33.902775336 (591h18m33.773858372s) [upload=518373]: Done starting upload
Dec 30 06:47:33.902944639 (591h18m33.774027674s) [upload=518373]: Uploading segments...
Dec 30 06:47:33.903006506 (591h18m33.774089541s) [upload=518373]: Got next segment. Inline: false
Dec 30 06:47:33.903036744 (591h18m33.774119779s) [upload=518373]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 30 06:47:33.903043197 (591h18m33.774126232s) [upload=518373]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 30 06:47:34.148006034 (591h18m34.019089069s) [seg_pos={0 0}, upload=518373]: Begin upload segment...
Dec 30 06:47:34.148062040 (591h18m34.019145075s) [seg_pos={0 0}, upload=518373]: Done begin upload segment.
Dec 30 06:47:34.148107336 (591h18m34.019190371s) [noise=true , address=45.91.116.5:4297 , piece_id=DG5N5IK35GLLLBJKWK7J53N3ER737OAGQBCHQ53AVMTIJHNWRXEA, seg_pos={0 0}, upload=518373]: Uploading piece...
Dec 30 06:47:34.148155579 (591h18m34.019238624s) [noise=true , address=145.79.227.46:4603 , piece_id=535JSH3VHKZETEICNVLFU3VCROFZVTZGOZRTN4O3UEIUDIQ6QEIQ, seg_pos={0 0}, upload=518373]: Uploading piece...
Dec 30 06:47:34.148683915 (591h18m34.01976695s) [noise=true , address=145.79.227.40:4051 , piece_id=N7JIQ4XFYDM4XQC4SMCUF2VYGGY35VA5BOKVCNAKD7BSKUSAPVNQ, seg_pos={0 0}, upload=518373]: Uploading piece...
Dec 30 06:47:34.148742016 (591h18m34.019825051s) [noise=true , address=145.79.227.48:4301 , piece_id=WMF7RPBJ3HUKXUJ63RLZFI6MVLUBJI3NLE4V7FS7NIG54XU5KLDA, seg_pos={0 0}, upload=518373]: Uploading piece...
Dec 30 06:47:34.149083746 (591h18m34.020166781s) [noise=true , address=45.91.116.4:4939 , piece_id=DRYXDGA2FTS2T5F47ZXZASU6O63V6LVCAVZPKK3OJTO5Y4IERHUQ, seg_pos={0 0}, upload=518373]: Uploading piece...
Dec 30 06:47:34.149391432 (591h18m34.020474477s) [noise=true , address=45.91.116.7:4010 , piece_id=TSCFP4MGODAU4IOWHIKWBKNNAE4G2ZGCBTVRKF3KVXCCP7Z6H7EQ, seg_pos={0 0}, upload=518373]: Uploading piece...
Dec 30 06:47:34.149687105 (591h18m34.02077014s) [noise=true , address=145.79.227.47:4265 , piece_id=FAVM24T5F2HBQIQS7IQ3GNVA5FNBCPZAUTJDVL5K4QYEGDI67MZQ, seg_pos={0 0}, upload=518373]: Uploading piece...
Dec 30 06:47:34.149987156 (591h18m34.021070191s) [noise=true , address=145.79.227.41:4085 , piece_id=V7X4XLDOB6JATX4JOWZ72XC7ONEDN64FIBBTZLJN2JLURNZRSH6Q, seg_pos={0 0}, upload=518373]: Uploading piece...
Dec 30 06:47:34.150321904 (591h18m34.021404949s) [noise=true , address=45.91.116.11:4120 , piece_id=CGPZDXJ44PHSBJ5NJUNKEUEFNYHUJPVWUETQGQ5HWAW3YZW4QDVA, seg_pos={0 0}, upload=518373]: Uploading piece...
Dec 30 06:47:34.150333786 (591h18m34.021416821s) [noise=true , address=145.79.227.39:4014 , piece_id=BLNHJN77DG2Y34Y3Z663ZUMQTBBOHWZKSW3QMBGLCVLNL6CEGJOA, seg_pos={0 0}, upload=518373]: Uploading piece...
Dec 30 06:47:34.150579524 (591h18m34.021662559s) [noise=true , address=145.79.227.19:4078 , piece_id=C2VTC5RDJBTU5Z2KKJENXLKYY2WIWBFELGDCRTJLRGSS3HWCG3XQ, seg_pos={0 0}, upload=518373]: Uploading piece...
Dec 30 06:47:34.150705945 (591h18m34.02178898s) [noise=true , address=145.79.227.32:4834 , piece_id=BQL2WD5B6ZKNH7B4AX7ELFSMKQVLVJCELGIDJYPKVSHURFJSRJLQ, seg_pos={0 0}, upload=518373]: Uploading piece...
Dec 30 06:47:34.151198954 (591h18m34.022281989s) [noise=true , address=45.91.116.14:4191 , piece_id=S6RREAK6WN5LM7T7TKBMPTDB7G2UWNGZ2L244ULRIMBEA6MOWGLA, seg_pos={0 0}, upload=518373]: Uploading piece...
Dec 30 06:47:34.151603264 (591h18m34.022686299s) [noise=true , address=145.79.227.43:4159 , piece_id=G55LMZZF32WFDATSY2BQKII65LYTA3IXEW5FIKYV444SNWQ7R2SQ, seg_pos={0 0}, upload=518373]: Uploading piece...
Dec 30 06:47:34.151984440 (591h18m34.023067485s) [noise=true , address=145.79.227.2:4114 , piece_id=QUHSCSX6TMOUGUKSUMAOENS4UBW7W4TBCTY4GAOCDH27IK2EQRBQ, seg_pos={0 0}, upload=518373]: Uploading piece...
Dec 30 06:47:34.152395842 (591h18m34.023478877s) [noise=true , address=145.79.227.17:4970 , piece_id=OR5LHPTUBOS3DDI4W2XFT5ED3Y5FGDOOVP6VPOHYYLELNS7SZB3A, seg_pos={0 0}, upload=518373]: Uploading piece...
Dec 30 06:47:35.844170154 (591h18m35.715253189s) [upload=518373]: Next returned nil segment
Dec 30 06:47:35.844218737 (591h18m35.715301792s) [upload=518373]: Waiting for error group managing segments...
Dec 30 06:47:36.127146455 (591h18m35.9982295s) [noise=true , address=145.79.227.19:4078 , piece_id=C2VTC5RDJBTU5Z2KKJENXLKYY2WIWBFELGDCRTJLRGSS3HWCG3XQ, seg_pos={0 0}, upload=518373]: Done uploading piece. err: <nil>
Dec 30 06:47:36.130047630 (591h18m36.001130665s) [noise=true , address=145.79.227.39:4014 , piece_id=BLNHJN77DG2Y34Y3Z663ZUMQTBBOHWZKSW3QMBGLCVLNL6CEGJOA, seg_pos={0 0}, upload=518373]: Done uploading piece. err: <nil>
Dec 30 06:47:36.140822898 (591h18m36.011905933s) [noise=true , address=145.79.227.2:4114 , piece_id=QUHSCSX6TMOUGUKSUMAOENS4UBW7W4TBCTY4GAOCDH27IK2EQRBQ, seg_pos={0 0}, upload=518373]: Done uploading piece. err: <nil>
Dec 30 06:47:36.141865904 (591h18m36.012948949s) [noise=true , address=145.79.227.46:4603 , piece_id=535JSH3VHKZETEICNVLFU3VCROFZVTZGOZRTN4O3UEIUDIQ6QEIQ, seg_pos={0 0}, upload=518373]: Done uploading piece. err: <nil>
Dec 30 06:47:36.142992949 (591h18m36.014075984s) [noise=true , address=145.79.227.48:4301 , piece_id=WMF7RPBJ3HUKXUJ63RLZFI6MVLUBJI3NLE4V7FS7NIG54XU5KLDA, seg_pos={0 0}, upload=518373]: Done uploading piece. err: <nil>
Dec 30 06:47:36.154997809 (591h18m36.026080844s) [noise=true , address=145.79.227.43:4159 , piece_id=G55LMZZF32WFDATSY2BQKII65LYTA3IXEW5FIKYV444SNWQ7R2SQ, seg_pos={0 0}, upload=518373]: Done uploading piece. err: <nil>
Dec 30 06:47:36.167251051 (591h18m36.038334096s) [noise=true , address=145.79.227.41:4085 , piece_id=V7X4XLDOB6JATX4JOWZ72XC7ONEDN64FIBBTZLJN2JLURNZRSH6Q, seg_pos={0 0}, upload=518373]: Done uploading piece. err: <nil>
Dec 30 06:47:36.171358592 (591h18m36.042441637s) [noise=true , address=145.79.227.47:4265 , piece_id=FAVM24T5F2HBQIQS7IQ3GNVA5FNBCPZAUTJDVL5K4QYEGDI67MZQ, seg_pos={0 0}, upload=518373]: Done uploading piece. err: <nil>
Dec 30 06:47:36.204427992 (591h18m36.075511027s) [noise=true , address=145.79.227.32:4834 , piece_id=BQL2WD5B6ZKNH7B4AX7ELFSMKQVLVJCELGIDJYPKVSHURFJSRJLQ, seg_pos={0 0}, upload=518373]: Done uploading piece. err: <nil>
Dec 30 06:47:36.219970897 (591h18m36.091053932s) [noise=true , address=145.79.227.17:4970 , piece_id=OR5LHPTUBOS3DDI4W2XFT5ED3Y5FGDOOVP6VPOHYYLELNS7SZB3A, seg_pos={0 0}, upload=518373]: Done uploading piece. err: <nil>
Dec 30 06:47:36.460845771 (591h18m36.331928816s) [noise=true , address=45.91.116.4:4939 , piece_id=DRYXDGA2FTS2T5F47ZXZASU6O63V6LVCAVZPKK3OJTO5Y4IERHUQ, seg_pos={0 0}, upload=518373]: Done uploading piece. err: <nil>
Dec 30 06:47:36.559892892 (591h18m36.430975927s) [noise=true , address=45.91.116.5:4297 , piece_id=DG5N5IK35GLLLBJKWK7J53N3ER737OAGQBCHQ53AVMTIJHNWRXEA, seg_pos={0 0}, upload=518373]: Done uploading piece. err: <nil>
Dec 30 06:47:36.564095394 (591h18m36.435178429s) [noise=true , address=45.91.116.7:4010 , piece_id=TSCFP4MGODAU4IOWHIKWBKNNAE4G2ZGCBTVRKF3KVXCCP7Z6H7EQ, seg_pos={0 0}, upload=518373]: Done uploading piece. err: <nil>
Dec 30 06:47:36.564160248 (591h18m36.435243283s) [seg_pos={0 0}, upload=518373]: Segment reached optimal threshold of 13 pieces.
Dec 30 06:47:36.564563686 (591h18m36.435646731s) [noise=true , address=45.91.116.11:4120 , piece_id=CGPZDXJ44PHSBJ5NJUNKEUEFNYHUJPVWUETQGQ5HWAW3YZW4QDVA, seg_pos={0 0}, upload=518373]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8): protocol: write tcp 188.166.225.170:50072->45.91.116.11:4120: use of closed network connection; write tcp 188.166.225.170:50072->45.91.116.11:4120: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50072->45.91.116.11:4120: use of closed network connection
Dec 30 06:47:36.564574417 (591h18m36.435657452s) [noise=true , address=45.91.116.11:4120 , piece_id=CGPZDXJ44PHSBJ5NJUNKEUEFNYHUJPVWUETQGQ5HWAW3YZW4QDVA, seg_pos={0 0}, upload=518373]: Long tail cancellation due to optimal threshold
Dec 30 06:47:36.564877924 (591h18m36.435960959s) [noise=true , address=45.91.116.14:4191 , piece_id=S6RREAK6WN5LM7T7TKBMPTDB7G2UWNGZ2L244ULRIMBEA6MOWGLA, seg_pos={0 0}, upload=518373]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq): protocol: expected piece hash; context canceled; EOF
Dec 30 06:47:36.564886651 (591h18m36.435969696s) [noise=true , address=45.91.116.14:4191 , piece_id=S6RREAK6WN5LM7T7TKBMPTDB7G2UWNGZ2L244ULRIMBEA6MOWGLA, seg_pos={0 0}, upload=518373]: Long tail cancellation due to optimal threshold
Dec 30 06:47:36.565006689 (591h18m36.436089734s) [noise=true , address=145.79.227.40:4051 , piece_id=N7JIQ4XFYDM4XQC4SMCUF2VYGGY35VA5BOKVCNAKD7BSKUSAPVNQ, seg_pos={0 0}, upload=518373]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rPbTARUrq7YsD8X75ocnyEvC517itk79s3Dp1wUg72fVuBS8n): protocol: expected piece hash; context canceled; EOF
Dec 30 06:47:36.565014714 (591h18m36.436097749s) [noise=true , address=145.79.227.40:4051 , piece_id=N7JIQ4XFYDM4XQC4SMCUF2VYGGY35VA5BOKVCNAKD7BSKUSAPVNQ, seg_pos={0 0}, upload=518373]: Long tail cancellation due to optimal threshold
Dec 30 06:47:36.565048769 (591h18m36.436131804s) [seg_pos={0 0}, upload=518373]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 30 06:47:36.565134443 (591h18m36.436217478s) [upload=518373]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 06:47:36.565140214 (591h18m36.436223259s) [upload=518373]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 06:47:36.856746186 (591h18m36.727829231s) [upload=518373]: Done uploading segments...
Dec 30 06:47:36.856766275 (591h18m36.72784932s) [upload=518373]: Upload finished. err: <nil>
Duration: 30.002365503s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
9 out of 60 is failed (threshold=6) [..........X..............XX...X......X..X..X.....X.....X....] duration:
3.547380863s trace:
https://jaeger.dev.storj.io/trace/2dfc5fc3a860afad
FAIL (2025-12-30 06:50:11 UTC)
orbiter warning: context deadline exceeded
Duration: 3.547406192s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)antiblip:
1 out of 60 is failed (threshold=3) [................................................X...........] duration:
6.902045153s trace:
https://jaeger.dev.storj.io/trace/4c2f4efaff0e0ea0
FAIL (2025-12-30 06:50:02 UTC)
orbiter warning:
operation: DeleteObject
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
load=518292]: Uploading piece...
Dec 30 06:41:13.598100152 (591h12m13.469183187s) [noise=true , address=81.171.14.70:4041 , piece_id=NORNA6WFTGPNOHU7MPTNNB5ONTBS5DJGCN2NWIZAEPYSUSIOSGEQ, seg_pos={0 0}, upload=518292]: Uploading piece...
Dec 30 06:41:13.598587971 (591h12m13.469671016s) [noise=true , address=89.149.201.147:4007 , piece_id=HZLSYGSBP4TJH5V5XF3Q3AETJU625ERNDBD3VXW2HUZTTPAHLBMA, seg_pos={0 0}, upload=518292]: Uploading piece...
Dec 30 06:41:13.599093143 (591h12m13.470176178s) [noise=true , address=89.149.201.147:4008 , piece_id=XYFRNJVIFFZA5S36KFPGZPVCPLRL2OW5QBJD3UR37VPPA2RX3RJA, seg_pos={0 0}, upload=518292]: Uploading piece...
Dec 30 06:41:13.599699999 (591h12m13.470783034s) [noise=true , address=81.171.14.70:4039 , piece_id=DEJDXFDHOSAMIP2FN5LJBJNBB4QLJSE2AUDW6IMA6IC4VRY7VAGQ, seg_pos={0 0}, upload=518292]: Uploading piece...
Dec 30 06:41:13.600136169 (591h12m13.471219204s) [noise=true , address=185.167.204.205:4034 , piece_id=OWZHQB5ZDT2QZ2QAHZJVRWQ222MX6CWLKJUOIAAGSMAVLWGJAOBA, seg_pos={0 0}, upload=518292]: Uploading piece...
Dec 30 06:41:13.600543184 (591h12m13.471626229s) [noise=true , address=185.167.204.206:4058 , piece_id=ROOCGK64M5RHIWNAVCSZNYTZ6M2ORWB7MJ27XJVDTFVQIKJZHY4A, seg_pos={0 0}, upload=518292]: Uploading piece...
Dec 30 06:41:13.601048286 (591h12m13.472131332s) [noise=true , address=81.171.14.70:4046 , piece_id=B2MTUNKMUP5S7XW4SVQ5QI2ZO3VFB5QVAKBBY4BIZIUILM6O7VYQ, seg_pos={0 0}, upload=518292]: Uploading piece...
Dec 30 06:41:13.601698784 (591h12m13.472781819s) [noise=true , address=185.167.204.206:4064 , piece_id=H4PP7HPHXRX2LOZ4776AMWAJDN7OC435E2K4BXI7XOZYKTB4YUNA, seg_pos={0 0}, upload=518292]: Uploading piece...
Dec 30 06:41:13.602168259 (591h12m13.473251304s) [noise=true , address=185.167.204.206:4065 , piece_id=LD3J4656Q5QEUCMJ5OKEYATEVRIKSXO7XGARTN25BVM7FDIYOWKA, seg_pos={0 0}, upload=518292]: Uploading piece...
Dec 30 06:41:13.602782708 (591h12m13.473865753s) [noise=true , address=81.171.14.70:4027 , piece_id=6WEORWX5QOMFSJUEADIIBHHIPKNFP4NJPZAJADBUTPUSG7MZJMOQ, seg_pos={0 0}, upload=518292]: Uploading piece...
Dec 30 06:41:14.423395002 (591h12m14.294478047s) [upload=518292]: Next returned nil segment
Dec 30 06:41:14.423414068 (591h12m14.294497103s) [upload=518292]: Waiting for error group managing segments...
Dec 30 06:41:14.817650316 (591h12m14.688733351s) [noise=true , address=185.167.204.205:4029 , piece_id=KUOMMYJ2F4NIK6VN4C5M3Y3TMHYEZH7QFNJ6K3CLOAVI5R6VKTHQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:14.817870054 (591h12m14.688953099s) [noise=true , address=185.167.204.205:4019 , piece_id=H6AH654VGX6NHBUKODIWGL7I2RKE3DDMY4UM6VDONP2EP3NPDJBA, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:14.818623698 (591h12m14.689706744s) [noise=true , address=185.167.204.206:4065 , piece_id=LD3J4656Q5QEUCMJ5OKEYATEVRIKSXO7XGARTN25BVM7FDIYOWKA, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:14.818932436 (591h12m14.690015481s) [noise=true , address=185.167.204.206:4061 , piece_id=T2XS4IFPQSVS3T45SPQ5ZDDVT2VMTMXHKL2ZMX4OIOPRR67SCD2Q, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:14.819147046 (591h12m14.690230091s) [noise=true , address=185.167.204.206:4071 , piece_id=QI2TOY5KCDZVWCCLMAJXVUWD2HB7S7IOTEAV2EHAMUNHKOFKTNYA, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:14.819678578 (591h12m14.690761623s) [noise=true , address=185.167.204.206:4059 , piece_id=LMOGTWX2YIRR3EKNROK43O2B27S5L5U6GYB7HBVRBPQPH4WDDWZQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:14.819905028 (591h12m14.690988063s) [noise=true , address=185.167.204.206:4058 , piece_id=ROOCGK64M5RHIWNAVCSZNYTZ6M2ORWB7MJ27XJVDTFVQIKJZHY4A, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:14.820479312 (591h12m14.691562357s) [noise=true , address=185.167.204.205:4021 , piece_id=KP7J5XSY4SKSKZG7XXFZMNKMRP4472PXI7CW2UQBZICN4MAFO5RQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:14.823149296 (591h12m14.694232331s) [noise=true , address=185.167.204.205:4034 , piece_id=OWZHQB5ZDT2QZ2QAHZJVRWQ222MX6CWLKJUOIAAGSMAVLWGJAOBA, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:14.827906996 (591h12m14.698990041s) [noise=true , address=185.167.204.206:4055 , piece_id=VYEDR6XHZBZMTVJ5U5OF67WZOGVMCMQLXXBQUI7DHBCCVQ2RRERQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:14.828003769 (591h12m14.699086814s) [noise=true , address=185.167.204.205:4016 , piece_id=I6MUDLAXUUDJL2C4HFE22C2L56A7D6HSCNWXKWJ7OKO2N4RA4W7Q, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:14.828893203 (591h12m14.699976238s) [noise=true , address=185.167.204.206:4064 , piece_id=H4PP7HPHXRX2LOZ4776AMWAJDN7OC435E2K4BXI7XOZYKTB4YUNA, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:14.829477074 (591h12m14.700560109s) [noise=true , address=185.167.204.205:4017 , piece_id=CZP65KETMGOHFDMV7KENRL4V36UHBPWRDCBBPUM5NE5L2PV4AQCQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:14.829774972 (591h12m14.700858007s) [noise=true , address=185.167.204.206:4043 , piece_id=G7VZYID4TXGNPDN63NQULXOVSX42WMFFNPBX6KQJ5L2WSW4D2YGQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:14.831734493 (591h12m14.702817528s) [noise=true , address=185.167.204.206:4049 , piece_id=BQ7JEGSGPIFJUGZFBYRPDMBNPZ24BG6TEG4VUM4JUBTE4T2RF4SA, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:15.405688561 (591h12m15.276771606s) [noise=true , address=81.171.14.70:4027 , piece_id=6WEORWX5QOMFSJUEADIIBHHIPKNFP4NJPZAJADBUTPUSG7MZJMOQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:15.417110440 (591h12m15.288193485s) [noise=true , address=81.171.14.70:4031 , piece_id=U6Z4SNK32ZSWQ7NDR7ACXGRC4BADDOPR65ILBR64QLGVNDBADFFQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:15.425600446 (591h12m15.296683481s) [noise=true , address=81.171.14.70:4037 , piece_id=6A25LEIAQEJB3DNCTXQT6K7ZSKIGOW4QUGMNKLOG3CT2XXH25PSA, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:15.499056228 (591h12m15.370139263s) [noise=true , address=81.171.14.70:4038 , piece_id=3WRHW3VGXWHOEOV5Q6CLYSF7MUVWS7QHTNHLSTCKAUERPHAH3I3Q, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:15.514998925 (591h12m15.38608197s) [noise=true , address=81.171.14.70:4041 , piece_id=NORNA6WFTGPNOHU7MPTNNB5ONTBS5DJGCN2NWIZAEPYSUSIOSGEQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:15.528579544 (591h12m15.399662589s) [noise=true , address=81.171.14.70:4030 , piece_id=KITOJIT34CHS2QZBXCW2SJPZB4RHEHCMJ6IYQ4Z5KLL24YAWBIUQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:15.528991398 (591h12m15.400074434s) [noise=true , address=81.171.14.70:4047 , piece_id=ZWV5DP4YC3MSYDWEQH4ZK3VR4WZ4DUTI4VL3WVLQKFIEKIW44LOQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:15.534844544 (591h12m15.405927579s) [noise=true , address=81.171.14.70:4035 , piece_id=XCLL5SOAPJSSGF5YYQH6ZKGOHMLS6RI6H26FOW5XIJRY3TWYNEGA, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:15.915614910 (591h12m15.786697945s) [noise=true , address=81.171.14.70:4039 , piece_id=DEJDXFDHOSAMIP2FN5LJBJNBB4QLJSE2AUDW6IMA6IC4VRY7VAGQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:15.967820176 (591h12m15.838903211s) [noise=true , address=81.171.14.70:4046 , piece_id=B2MTUNKMUP5S7XW4SVQ5QI2ZO3VFB5QVAKBBY4BIZIUILM6O7VYQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:16.172116128 (591h12m16.043199163s) [noise=true , address=89.149.201.147:4007 , piece_id=HZLSYGSBP4TJH5V5XF3Q3AETJU625ERNDBD3VXW2HUZTTPAHLBMA, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:16.172137559 (591h12m16.043220604s) [seg_pos={0 0}, upload=518292]: Segment reached optimal threshold of 26 pieces.
Dec 30 06:41:16.172336949 (591h12m16.043419994s) [noise=true , address=89.149.201.147:4008 , piece_id=XYFRNJVIFFZA5S36KFPGZPVCPLRL2OW5QBJD3UR37VPPA2RX3RJA, seg_pos={0 0}, upload=518292]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QN8VP6o7qGw9Dd8EafU9xe7s7Cuf1FyqPMbnRaznrfPxfEUUn): protocol: expected piece hash; context canceled; EOF
Dec 30 06:41:16.172344213 (591h12m16.043427258s) [noise=true , address=89.149.201.147:4008 , piece_id=XYFRNJVIFFZA5S36KFPGZPVCPLRL2OW5QBJD3UR37VPPA2RX3RJA, seg_pos={0 0}, upload=518292]: Long tail cancellation due to optimal threshold
Dec 30 06:41:16.172554002 (591h12m16.043637037s) [noise=true , address=89.149.201.147:4020 , piece_id=N7G5DGEDYVKBTO5S6BYW3IYEOK7HAMK5JQIBMVPMPYRDHXODWEGA, seg_pos={0 0}, upload=518292]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12YeKHJH7CoouLFJYENJhzVLNkA32hJaqjLz3dXhr78eaSGtcic): protocol: expected piece hash; context canceled; EOF
Dec 30 06:41:16.172565233 (591h12m16.043648278s) [noise=true , address=89.149.201.147:4020 , piece_id=N7G5DGEDYVKBTO5S6BYW3IYEOK7HAMK5JQIBMVPMPYRDHXODWEGA, seg_pos={0 0}, upload=518292]: Long tail cancellation due to optimal threshold
Dec 30 06:41:16.172587506 (591h12m16.043670541s) [seg_pos={0 0}, upload=518292]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil>
Dec 30 06:41:16.172717945 (591h12m16.04380098s) [upload=518292]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 06:41:16.172724567 (591h12m16.043807602s) [upload=518292]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 06:41:16.404092237 (591h12m16.275175292s) [upload=518292]: Done uploading segments...
Dec 30 06:41:16.404119940 (591h12m16.275202975s) [upload=518292]: Upload finished. err: <nil>
Duration: 6.902076543s
core
uplink
eu-select-1
eu1-placement-15
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
4 out of 60 is failed (threshold=3) [X...X......X............X...................................] duration:
14.628108321s trace:
https://jaeger.dev.storj.io/trace/45b769fc71e13448
FAIL (2025-12-30 06:49:56 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: context deadline exceeded
======LOGS======
.577697683 (590h53m11.448780728s) [noise=true , address=193.149.164.143:4704 , piece_id=4T2IJIA6WHI5LMIUP5BLJLCD3GRY3A6PXR6G5OWJWUSYZT5ELGHQ, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.578173528 (590h53m11.449256573s) [noise=true , address=64.49.13.27:4965 , piece_id=CO4O5F2IBZAAKBZBV3HWNNOXZOGLNNDPALWMQZ3DEM3MREASVFEA, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.578542992 (590h53m11.449626027s) [noise=true , address=77.90.31.15:4729 , piece_id=NVMXFEILHGT622IIPYYT4FFHC2WZLHMCZM4VT5WUPHRAPH3DX6TA, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.578846911 (590h53m11.449929946s) [noise=true , address=145.79.227.43:4177 , piece_id=YM37ZI3WNZ7SXGTFC224TOVVTHYNFIPRQB2TFLEZ735RNZE553FQ, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.579458044 (590h53m11.450541089s) [noise=true , address=45.91.116.11:4150 , piece_id=E66IDHTJDLYLYJIOWMOIKOOWLXCUNCMBIQ7SQBAD3VTJF6VX5JLQ, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.579891209 (590h53m11.450974245s) [noise=true , address=145.79.227.29:4945 , piece_id=SHL73IDEI54GRCWYNWHRUSJZZ4UUXREAX54DFJLTG5QPZCF5QKIQ, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.579915535 (590h53m11.45099858s) [noise=true , address=145.79.227.30:4772 , piece_id=PVMD5DWRRPVARTSZ6O7HN6MUVTZUYWLYSNQR5GDMNKUJZGXMYPFQ, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.580325466 (590h53m11.451408511s) [noise=true , address=77.90.31.12:4603 , piece_id=2I4L7F2WEGK5OSLODJSJ4672URBA7DUGXWZKTYB6ZNA2TQIGVXWA, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.580379599 (590h53m11.451462644s) [noise=true , address=77.90.31.14:4671 , piece_id=263TRISJ5ADFTX43ULSSGBAF2KMOXAPISJ56LLE53D74XOEMQ5EA, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.580692044 (590h53m11.451775089s) [noise=true , address=145.79.227.3:4175 , piece_id=UCV5WQNVWEAMEDBWDNXEVSQEGXPJE7KD6YSWC5A4WXYUKUCK4NLA, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.580857859 (590h53m11.451940894s) [noise=true , address=77.90.31.19:4860 , piece_id=U2MNFOAD7FHCDBE4FUYOROIP3P33ROJ7K75SF5Z747EFKFTF7RGA, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.581089210 (590h53m11.452172255s) [noise=true , address=145.79.227.44:4203 , piece_id=WHCKQ2FRLYAZE7Q6HYZNTS4EISVHZNAPDDKH34WN33ALK4MVK2EA, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.581351320 (590h53m11.452434355s) [noise=true , address=64.49.13.154:4928 , piece_id=SEVBZBMNGURP5TRUEBA4PHRSA6MNANVJUKINALJUWPQWBGKVYNTA, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.581457091 (590h53m11.452540126s) [noise=true , address=145.79.227.34:4909 , piece_id=3QSWTNR5LGUUAY3N5HJKSPL3TRQ6WO5F5OD4VTXKWHWUXUF7PVVQ, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.581863144 (590h53m11.452946179s) [noise=true , address=145.79.227.33:4881 , piece_id=EX66JF7U7A4ZSRWJMP4MCLQRPW2ZZFZFIUNV72XZXU7E5O3KWX3A, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.584010443 (590h53m11.455093478s) [noise=true , address=145.79.227.40:4070 , piece_id=YOAGDXNN4TTCK35LLQDSP24BOWLZXGNSQU7WMKEEVVMPXDVI74JA, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.585122791 (590h53m11.456205826s) [noise=true , address=45.91.116.14:4217 , piece_id=L4MAHTGKTNWBSN3YTB74E3DBJWVMA3DPU57MQK5FNK35DXRZ6AIA, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:12.983305402 (590h53m12.854388447s) [upload=517998]: Next returned nil segment
Dec 30 06:22:12.983321313 (590h53m12.854404348s) [upload=517998]: Waiting for error group managing segments...
Dec 30 06:22:13.379703053 (590h53m13.250786088s) [noise=true , address=64.49.13.27:4965 , piece_id=CO4O5F2IBZAAKBZBV3HWNNOXZOGLNNDPALWMQZ3DEM3MREASVFEA, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:13.418055622 (590h53m13.289138657s) [noise=true , address=145.79.227.3:4175 , piece_id=UCV5WQNVWEAMEDBWDNXEVSQEGXPJE7KD6YSWC5A4WXYUKUCK4NLA, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:13.509034160 (590h53m13.380117205s) [noise=true , address=64.49.13.154:4928 , piece_id=SEVBZBMNGURP5TRUEBA4PHRSA6MNANVJUKINALJUWPQWBGKVYNTA, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:13.642886154 (590h53m13.513969199s) [noise=true , address=77.90.31.14:4671 , piece_id=263TRISJ5ADFTX43ULSSGBAF2KMOXAPISJ56LLE53D74XOEMQ5EA, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:13.788145520 (590h53m13.659228555s) [noise=true , address=77.90.31.12:4603 , piece_id=2I4L7F2WEGK5OSLODJSJ4672URBA7DUGXWZKTYB6ZNA2TQIGVXWA, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:13.858878757 (590h53m13.729961792s) [noise=true , address=145.79.227.49:4369 , piece_id=XUEXGNJJIJ4DUBDNUV7CKF6J5JYHSLZAEPVGEDGBUBY42BIP4XEQ, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:13.879550569 (590h53m13.750633614s) [noise=true , address=145.79.227.34:4909 , piece_id=3QSWTNR5LGUUAY3N5HJKSPL3TRQ6WO5F5OD4VTXKWHWUXUF7PVVQ, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:13.940897723 (590h53m13.811980769s) [noise=true , address=77.90.31.19:4860 , piece_id=U2MNFOAD7FHCDBE4FUYOROIP3P33ROJ7K75SF5Z747EFKFTF7RGA, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:13.989791936 (590h53m13.860874981s) [noise=true , address=45.91.116.11:4150 , piece_id=E66IDHTJDLYLYJIOWMOIKOOWLXCUNCMBIQ7SQBAD3VTJF6VX5JLQ, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:14.289516718 (590h53m14.160599753s) [noise=true , address=145.79.227.44:4203 , piece_id=WHCKQ2FRLYAZE7Q6HYZNTS4EISVHZNAPDDKH34WN33ALK4MVK2EA, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:14.338651537 (590h53m14.209734582s) [noise=true , address=77.90.31.15:4729 , piece_id=NVMXFEILHGT622IIPYYT4FFHC2WZLHMCZM4VT5WUPHRAPH3DX6TA, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:14.344145168 (590h53m14.215228213s) [noise=true , address=145.79.227.30:4772 , piece_id=PVMD5DWRRPVARTSZ6O7HN6MUVTZUYWLYSNQR5GDMNKUJZGXMYPFQ, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:14.667232847 (590h53m14.538315892s) [noise=true , address=193.149.164.143:4704 , piece_id=4T2IJIA6WHI5LMIUP5BLJLCD3GRY3A6PXR6G5OWJWUSYZT5ELGHQ, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:14.671388990 (590h53m14.542472025s) [noise=true , address=145.79.227.29:4945 , piece_id=SHL73IDEI54GRCWYNWHRUSJZZ4UUXREAX54DFJLTG5QPZCF5QKIQ, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:14.671415190 (590h53m14.542498226s) [seg_pos={0 0}, upload=517998]: Segment reached optimal threshold of 14 pieces.
Dec 30 06:22:14.671803600 (590h53m14.542886655s) [noise=true , address=145.79.227.40:4070 , piece_id=YOAGDXNN4TTCK35LLQDSP24BOWLZXGNSQU7WMKEEVVMPXDVI74JA, seg_pos={0 0}, upload=517998]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12hU1LwL2gX7rVnrNgzaBCYB37ziqf7s7aUgtF2Uc3antReQ6me): protocol: expected piece hash; context canceled; EOF
Dec 30 06:22:14.671814470 (590h53m14.542897525s) [noise=true , address=145.79.227.40:4070 , piece_id=YOAGDXNN4TTCK35LLQDSP24BOWLZXGNSQU7WMKEEVVMPXDVI74JA, seg_pos={0 0}, upload=517998]: Long tail cancellation due to optimal threshold
Dec 30 06:22:14.672014932 (590h53m14.543097977s) [noise=true , address=145.79.227.43:4177 , piece_id=YM37ZI3WNZ7SXGTFC224TOVVTHYNFIPRQB2TFLEZ735RNZE553FQ, seg_pos={0 0}, upload=517998]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bhN3cR22QuAYihnnB3x3R97UZRWJ8sFPP8fjQcXZ7qhNZphZG): protocol: expected piece hash; context canceled; EOF
Dec 30 06:22:14.672022637 (590h53m14.543105672s) [noise=true , address=145.79.227.43:4177 , piece_id=YM37ZI3WNZ7SXGTFC224TOVVTHYNFIPRQB2TFLEZ735RNZE553FQ, seg_pos={0 0}, upload=517998]: Long tail cancellation due to optimal threshold
Dec 30 06:22:14.672088933 (590h53m14.543171988s) [noise=true , address=145.79.227.33:4881 , piece_id=EX66JF7U7A4ZSRWJMP4MCLQRPW2ZZFZFIUNV72XZXU7E5O3KWX3A, seg_pos={0 0}, upload=517998]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mMBjg1b4MqTZPE5NpNNMYDzmsHqWYotE5jBNmZLyA5FyYLZRp): protocol: expected piece hash; context canceled; EOF
Dec 30 06:22:14.672127677 (590h53m14.543210742s) [noise=true , address=145.79.227.33:4881 , piece_id=EX66JF7U7A4ZSRWJMP4MCLQRPW2ZZFZFIUNV72XZXU7E5O3KWX3A, seg_pos={0 0}, upload=517998]: Long tail cancellation due to optimal threshold
Dec 30 06:22:14.672312739 (590h53m14.543395774s) [noise=true , address=45.91.116.14:4217 , piece_id=L4MAHTGKTNWBSN3YTB74E3DBJWVMA3DPU57MQK5FNK35DXRZ6AIA, seg_pos={0 0}, upload=517998]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xSXWFiq27Dd2vBDD9vVnhDHVG59UmDKn3hVAhf3oyo5goVNaE): protocol: expected piece hash; context canceled; EOF
Dec 30 06:22:14.672319953 (590h53m14.543402988s) [noise=true , address=45.91.116.14:4217 , piece_id=L4MAHTGKTNWBSN3YTB74E3DBJWVMA3DPU57MQK5FNK35DXRZ6AIA, seg_pos={0 0}, upload=517998]: Long tail cancellation due to optimal threshold
Dec 30 06:22:14.672337847 (590h53m14.543420882s) [seg_pos={0 0}, upload=517998]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 30 06:22:14.672452947 (590h53m14.543535982s) [upload=517998]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 06:22:14.672458507 (590h53m14.543541542s) [upload=517998]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 06:22:14.981437069 (590h53m14.852520114s) [upload=517998]: Done uploading segments...
Dec 30 06:22:14.981456546 (590h53m14.852539581s) [upload=517998]: Upload finished. err: <nil>
Duration: 14.628139189s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.........X..................................................] duration:
9.069627688s trace:
https://jaeger.dev.storj.io/trace/deb96436003dee
FAIL (2025-12-30 06:49:56 UTC)
orbiter warning: read tcp 188.166.225.170:52458->45.38.24.2:443: read: connection reset by peer
Duration: 9.069696419s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
9 out of 60 is failed (threshold=6) [...........X..............XX...X......X..X..X.....X.....X...] duration:
3.373867784s trace:
https://jaeger.dev.storj.io/trace/2dfc5fc3a860afad
FAIL (2025-12-30 06:49:41 UTC)
orbiter warning: context deadline exceeded
Duration: 3.373895588s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
4 out of 60 is failed (threshold=3) [.X...X......X............X..................................] duration:
42.349998211s trace:
https://jaeger.dev.storj.io/trace/45b769fc71e13448
FAIL (2025-12-30 06:49:40 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: context deadline exceeded
======LOGS======
.577697683 (590h53m11.448780728s) [noise=true , address=193.149.164.143:4704 , piece_id=4T2IJIA6WHI5LMIUP5BLJLCD3GRY3A6PXR6G5OWJWUSYZT5ELGHQ, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.578173528 (590h53m11.449256573s) [noise=true , address=64.49.13.27:4965 , piece_id=CO4O5F2IBZAAKBZBV3HWNNOXZOGLNNDPALWMQZ3DEM3MREASVFEA, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.578542992 (590h53m11.449626027s) [noise=true , address=77.90.31.15:4729 , piece_id=NVMXFEILHGT622IIPYYT4FFHC2WZLHMCZM4VT5WUPHRAPH3DX6TA, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.578846911 (590h53m11.449929946s) [noise=true , address=145.79.227.43:4177 , piece_id=YM37ZI3WNZ7SXGTFC224TOVVTHYNFIPRQB2TFLEZ735RNZE553FQ, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.579458044 (590h53m11.450541089s) [noise=true , address=45.91.116.11:4150 , piece_id=E66IDHTJDLYLYJIOWMOIKOOWLXCUNCMBIQ7SQBAD3VTJF6VX5JLQ, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.579891209 (590h53m11.450974245s) [noise=true , address=145.79.227.29:4945 , piece_id=SHL73IDEI54GRCWYNWHRUSJZZ4UUXREAX54DFJLTG5QPZCF5QKIQ, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.579915535 (590h53m11.45099858s) [noise=true , address=145.79.227.30:4772 , piece_id=PVMD5DWRRPVARTSZ6O7HN6MUVTZUYWLYSNQR5GDMNKUJZGXMYPFQ, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.580325466 (590h53m11.451408511s) [noise=true , address=77.90.31.12:4603 , piece_id=2I4L7F2WEGK5OSLODJSJ4672URBA7DUGXWZKTYB6ZNA2TQIGVXWA, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.580379599 (590h53m11.451462644s) [noise=true , address=77.90.31.14:4671 , piece_id=263TRISJ5ADFTX43ULSSGBAF2KMOXAPISJ56LLE53D74XOEMQ5EA, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.580692044 (590h53m11.451775089s) [noise=true , address=145.79.227.3:4175 , piece_id=UCV5WQNVWEAMEDBWDNXEVSQEGXPJE7KD6YSWC5A4WXYUKUCK4NLA, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.580857859 (590h53m11.451940894s) [noise=true , address=77.90.31.19:4860 , piece_id=U2MNFOAD7FHCDBE4FUYOROIP3P33ROJ7K75SF5Z747EFKFTF7RGA, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.581089210 (590h53m11.452172255s) [noise=true , address=145.79.227.44:4203 , piece_id=WHCKQ2FRLYAZE7Q6HYZNTS4EISVHZNAPDDKH34WN33ALK4MVK2EA, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.581351320 (590h53m11.452434355s) [noise=true , address=64.49.13.154:4928 , piece_id=SEVBZBMNGURP5TRUEBA4PHRSA6MNANVJUKINALJUWPQWBGKVYNTA, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.581457091 (590h53m11.452540126s) [noise=true , address=145.79.227.34:4909 , piece_id=3QSWTNR5LGUUAY3N5HJKSPL3TRQ6WO5F5OD4VTXKWHWUXUF7PVVQ, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.581863144 (590h53m11.452946179s) [noise=true , address=145.79.227.33:4881 , piece_id=EX66JF7U7A4ZSRWJMP4MCLQRPW2ZZFZFIUNV72XZXU7E5O3KWX3A, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.584010443 (590h53m11.455093478s) [noise=true , address=145.79.227.40:4070 , piece_id=YOAGDXNN4TTCK35LLQDSP24BOWLZXGNSQU7WMKEEVVMPXDVI74JA, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.585122791 (590h53m11.456205826s) [noise=true , address=45.91.116.14:4217 , piece_id=L4MAHTGKTNWBSN3YTB74E3DBJWVMA3DPU57MQK5FNK35DXRZ6AIA, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:12.983305402 (590h53m12.854388447s) [upload=517998]: Next returned nil segment
Dec 30 06:22:12.983321313 (590h53m12.854404348s) [upload=517998]: Waiting for error group managing segments...
Dec 30 06:22:13.379703053 (590h53m13.250786088s) [noise=true , address=64.49.13.27:4965 , piece_id=CO4O5F2IBZAAKBZBV3HWNNOXZOGLNNDPALWMQZ3DEM3MREASVFEA, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:13.418055622 (590h53m13.289138657s) [noise=true , address=145.79.227.3:4175 , piece_id=UCV5WQNVWEAMEDBWDNXEVSQEGXPJE7KD6YSWC5A4WXYUKUCK4NLA, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:13.509034160 (590h53m13.380117205s) [noise=true , address=64.49.13.154:4928 , piece_id=SEVBZBMNGURP5TRUEBA4PHRSA6MNANVJUKINALJUWPQWBGKVYNTA, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:13.642886154 (590h53m13.513969199s) [noise=true , address=77.90.31.14:4671 , piece_id=263TRISJ5ADFTX43ULSSGBAF2KMOXAPISJ56LLE53D74XOEMQ5EA, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:13.788145520 (590h53m13.659228555s) [noise=true , address=77.90.31.12:4603 , piece_id=2I4L7F2WEGK5OSLODJSJ4672URBA7DUGXWZKTYB6ZNA2TQIGVXWA, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:13.858878757 (590h53m13.729961792s) [noise=true , address=145.79.227.49:4369 , piece_id=XUEXGNJJIJ4DUBDNUV7CKF6J5JYHSLZAEPVGEDGBUBY42BIP4XEQ, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:13.879550569 (590h53m13.750633614s) [noise=true , address=145.79.227.34:4909 , piece_id=3QSWTNR5LGUUAY3N5HJKSPL3TRQ6WO5F5OD4VTXKWHWUXUF7PVVQ, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:13.940897723 (590h53m13.811980769s) [noise=true , address=77.90.31.19:4860 , piece_id=U2MNFOAD7FHCDBE4FUYOROIP3P33ROJ7K75SF5Z747EFKFTF7RGA, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:13.989791936 (590h53m13.860874981s) [noise=true , address=45.91.116.11:4150 , piece_id=E66IDHTJDLYLYJIOWMOIKOOWLXCUNCMBIQ7SQBAD3VTJF6VX5JLQ, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:14.289516718 (590h53m14.160599753s) [noise=true , address=145.79.227.44:4203 , piece_id=WHCKQ2FRLYAZE7Q6HYZNTS4EISVHZNAPDDKH34WN33ALK4MVK2EA, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:14.338651537 (590h53m14.209734582s) [noise=true , address=77.90.31.15:4729 , piece_id=NVMXFEILHGT622IIPYYT4FFHC2WZLHMCZM4VT5WUPHRAPH3DX6TA, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:14.344145168 (590h53m14.215228213s) [noise=true , address=145.79.227.30:4772 , piece_id=PVMD5DWRRPVARTSZ6O7HN6MUVTZUYWLYSNQR5GDMNKUJZGXMYPFQ, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:14.667232847 (590h53m14.538315892s) [noise=true , address=193.149.164.143:4704 , piece_id=4T2IJIA6WHI5LMIUP5BLJLCD3GRY3A6PXR6G5OWJWUSYZT5ELGHQ, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:14.671388990 (590h53m14.542472025s) [noise=true , address=145.79.227.29:4945 , piece_id=SHL73IDEI54GRCWYNWHRUSJZZ4UUXREAX54DFJLTG5QPZCF5QKIQ, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:14.671415190 (590h53m14.542498226s) [seg_pos={0 0}, upload=517998]: Segment reached optimal threshold of 14 pieces.
Dec 30 06:22:14.671803600 (590h53m14.542886655s) [noise=true , address=145.79.227.40:4070 , piece_id=YOAGDXNN4TTCK35LLQDSP24BOWLZXGNSQU7WMKEEVVMPXDVI74JA, seg_pos={0 0}, upload=517998]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12hU1LwL2gX7rVnrNgzaBCYB37ziqf7s7aUgtF2Uc3antReQ6me): protocol: expected piece hash; context canceled; EOF
Dec 30 06:22:14.671814470 (590h53m14.542897525s) [noise=true , address=145.79.227.40:4070 , piece_id=YOAGDXNN4TTCK35LLQDSP24BOWLZXGNSQU7WMKEEVVMPXDVI74JA, seg_pos={0 0}, upload=517998]: Long tail cancellation due to optimal threshold
Dec 30 06:22:14.672014932 (590h53m14.543097977s) [noise=true , address=145.79.227.43:4177 , piece_id=YM37ZI3WNZ7SXGTFC224TOVVTHYNFIPRQB2TFLEZ735RNZE553FQ, seg_pos={0 0}, upload=517998]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bhN3cR22QuAYihnnB3x3R97UZRWJ8sFPP8fjQcXZ7qhNZphZG): protocol: expected piece hash; context canceled; EOF
Dec 30 06:22:14.672022637 (590h53m14.543105672s) [noise=true , address=145.79.227.43:4177 , piece_id=YM37ZI3WNZ7SXGTFC224TOVVTHYNFIPRQB2TFLEZ735RNZE553FQ, seg_pos={0 0}, upload=517998]: Long tail cancellation due to optimal threshold
Dec 30 06:22:14.672088933 (590h53m14.543171988s) [noise=true , address=145.79.227.33:4881 , piece_id=EX66JF7U7A4ZSRWJMP4MCLQRPW2ZZFZFIUNV72XZXU7E5O3KWX3A, seg_pos={0 0}, upload=517998]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mMBjg1b4MqTZPE5NpNNMYDzmsHqWYotE5jBNmZLyA5FyYLZRp): protocol: expected piece hash; context canceled; EOF
Dec 30 06:22:14.672127677 (590h53m14.543210742s) [noise=true , address=145.79.227.33:4881 , piece_id=EX66JF7U7A4ZSRWJMP4MCLQRPW2ZZFZFIUNV72XZXU7E5O3KWX3A, seg_pos={0 0}, upload=517998]: Long tail cancellation due to optimal threshold
Dec 30 06:22:14.672312739 (590h53m14.543395774s) [noise=true , address=45.91.116.14:4217 , piece_id=L4MAHTGKTNWBSN3YTB74E3DBJWVMA3DPU57MQK5FNK35DXRZ6AIA, seg_pos={0 0}, upload=517998]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xSXWFiq27Dd2vBDD9vVnhDHVG59UmDKn3hVAhf3oyo5goVNaE): protocol: expected piece hash; context canceled; EOF
Dec 30 06:22:14.672319953 (590h53m14.543402988s) [noise=true , address=45.91.116.14:4217 , piece_id=L4MAHTGKTNWBSN3YTB74E3DBJWVMA3DPU57MQK5FNK35DXRZ6AIA, seg_pos={0 0}, upload=517998]: Long tail cancellation due to optimal threshold
Dec 30 06:22:14.672337847 (590h53m14.543420882s) [seg_pos={0 0}, upload=517998]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 30 06:22:14.672452947 (590h53m14.543535982s) [upload=517998]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 06:22:14.672458507 (590h53m14.543541542s) [upload=517998]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 06:22:14.981437069 (590h53m14.852520114s) [upload=517998]: Done uploading segments...
Dec 30 06:22:14.981456546 (590h53m14.852539581s) [upload=517998]: Upload finished. err: <nil>
Duration: 42.350030802s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [..........X.................................................] duration:
5.859315229s trace:
https://jaeger.dev.storj.io/trace/deb96436003dee
FAIL (2025-12-30 06:49:33 UTC)
orbiter warning: read tcp 188.166.225.170:52458->45.38.24.2:443: read: connection reset by peer
Duration: 5.859334174s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
10 out of 60 is failed (threshold=6) [X...........X..............XX...X......X..X..X.....X.....X..] duration:
3.620341681s trace:
https://jaeger.dev.storj.io/trace/2dfc5fc3a860afad
FAIL (2025-12-30 06:49:10 UTC)
orbiter warning: context deadline exceeded
Duration: 3.620376196s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)antiblip:
1 out of 60 is failed (threshold=3) [.................................................X..........] duration:
7.07657995s trace:
https://jaeger.dev.storj.io/trace/4c2f4efaff0e0ea0
FAIL (2025-12-30 06:49:04 UTC)
orbiter warning:
operation: DeleteObject
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
load=518292]: Uploading piece...
Dec 30 06:41:13.598100152 (591h12m13.469183187s) [noise=true , address=81.171.14.70:4041 , piece_id=NORNA6WFTGPNOHU7MPTNNB5ONTBS5DJGCN2NWIZAEPYSUSIOSGEQ, seg_pos={0 0}, upload=518292]: Uploading piece...
Dec 30 06:41:13.598587971 (591h12m13.469671016s) [noise=true , address=89.149.201.147:4007 , piece_id=HZLSYGSBP4TJH5V5XF3Q3AETJU625ERNDBD3VXW2HUZTTPAHLBMA, seg_pos={0 0}, upload=518292]: Uploading piece...
Dec 30 06:41:13.599093143 (591h12m13.470176178s) [noise=true , address=89.149.201.147:4008 , piece_id=XYFRNJVIFFZA5S36KFPGZPVCPLRL2OW5QBJD3UR37VPPA2RX3RJA, seg_pos={0 0}, upload=518292]: Uploading piece...
Dec 30 06:41:13.599699999 (591h12m13.470783034s) [noise=true , address=81.171.14.70:4039 , piece_id=DEJDXFDHOSAMIP2FN5LJBJNBB4QLJSE2AUDW6IMA6IC4VRY7VAGQ, seg_pos={0 0}, upload=518292]: Uploading piece...
Dec 30 06:41:13.600136169 (591h12m13.471219204s) [noise=true , address=185.167.204.205:4034 , piece_id=OWZHQB5ZDT2QZ2QAHZJVRWQ222MX6CWLKJUOIAAGSMAVLWGJAOBA, seg_pos={0 0}, upload=518292]: Uploading piece...
Dec 30 06:41:13.600543184 (591h12m13.471626229s) [noise=true , address=185.167.204.206:4058 , piece_id=ROOCGK64M5RHIWNAVCSZNYTZ6M2ORWB7MJ27XJVDTFVQIKJZHY4A, seg_pos={0 0}, upload=518292]: Uploading piece...
Dec 30 06:41:13.601048286 (591h12m13.472131332s) [noise=true , address=81.171.14.70:4046 , piece_id=B2MTUNKMUP5S7XW4SVQ5QI2ZO3VFB5QVAKBBY4BIZIUILM6O7VYQ, seg_pos={0 0}, upload=518292]: Uploading piece...
Dec 30 06:41:13.601698784 (591h12m13.472781819s) [noise=true , address=185.167.204.206:4064 , piece_id=H4PP7HPHXRX2LOZ4776AMWAJDN7OC435E2K4BXI7XOZYKTB4YUNA, seg_pos={0 0}, upload=518292]: Uploading piece...
Dec 30 06:41:13.602168259 (591h12m13.473251304s) [noise=true , address=185.167.204.206:4065 , piece_id=LD3J4656Q5QEUCMJ5OKEYATEVRIKSXO7XGARTN25BVM7FDIYOWKA, seg_pos={0 0}, upload=518292]: Uploading piece...
Dec 30 06:41:13.602782708 (591h12m13.473865753s) [noise=true , address=81.171.14.70:4027 , piece_id=6WEORWX5QOMFSJUEADIIBHHIPKNFP4NJPZAJADBUTPUSG7MZJMOQ, seg_pos={0 0}, upload=518292]: Uploading piece...
Dec 30 06:41:14.423395002 (591h12m14.294478047s) [upload=518292]: Next returned nil segment
Dec 30 06:41:14.423414068 (591h12m14.294497103s) [upload=518292]: Waiting for error group managing segments...
Dec 30 06:41:14.817650316 (591h12m14.688733351s) [noise=true , address=185.167.204.205:4029 , piece_id=KUOMMYJ2F4NIK6VN4C5M3Y3TMHYEZH7QFNJ6K3CLOAVI5R6VKTHQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:14.817870054 (591h12m14.688953099s) [noise=true , address=185.167.204.205:4019 , piece_id=H6AH654VGX6NHBUKODIWGL7I2RKE3DDMY4UM6VDONP2EP3NPDJBA, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:14.818623698 (591h12m14.689706744s) [noise=true , address=185.167.204.206:4065 , piece_id=LD3J4656Q5QEUCMJ5OKEYATEVRIKSXO7XGARTN25BVM7FDIYOWKA, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:14.818932436 (591h12m14.690015481s) [noise=true , address=185.167.204.206:4061 , piece_id=T2XS4IFPQSVS3T45SPQ5ZDDVT2VMTMXHKL2ZMX4OIOPRR67SCD2Q, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:14.819147046 (591h12m14.690230091s) [noise=true , address=185.167.204.206:4071 , piece_id=QI2TOY5KCDZVWCCLMAJXVUWD2HB7S7IOTEAV2EHAMUNHKOFKTNYA, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:14.819678578 (591h12m14.690761623s) [noise=true , address=185.167.204.206:4059 , piece_id=LMOGTWX2YIRR3EKNROK43O2B27S5L5U6GYB7HBVRBPQPH4WDDWZQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:14.819905028 (591h12m14.690988063s) [noise=true , address=185.167.204.206:4058 , piece_id=ROOCGK64M5RHIWNAVCSZNYTZ6M2ORWB7MJ27XJVDTFVQIKJZHY4A, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:14.820479312 (591h12m14.691562357s) [noise=true , address=185.167.204.205:4021 , piece_id=KP7J5XSY4SKSKZG7XXFZMNKMRP4472PXI7CW2UQBZICN4MAFO5RQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:14.823149296 (591h12m14.694232331s) [noise=true , address=185.167.204.205:4034 , piece_id=OWZHQB5ZDT2QZ2QAHZJVRWQ222MX6CWLKJUOIAAGSMAVLWGJAOBA, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:14.827906996 (591h12m14.698990041s) [noise=true , address=185.167.204.206:4055 , piece_id=VYEDR6XHZBZMTVJ5U5OF67WZOGVMCMQLXXBQUI7DHBCCVQ2RRERQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:14.828003769 (591h12m14.699086814s) [noise=true , address=185.167.204.205:4016 , piece_id=I6MUDLAXUUDJL2C4HFE22C2L56A7D6HSCNWXKWJ7OKO2N4RA4W7Q, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:14.828893203 (591h12m14.699976238s) [noise=true , address=185.167.204.206:4064 , piece_id=H4PP7HPHXRX2LOZ4776AMWAJDN7OC435E2K4BXI7XOZYKTB4YUNA, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:14.829477074 (591h12m14.700560109s) [noise=true , address=185.167.204.205:4017 , piece_id=CZP65KETMGOHFDMV7KENRL4V36UHBPWRDCBBPUM5NE5L2PV4AQCQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:14.829774972 (591h12m14.700858007s) [noise=true , address=185.167.204.206:4043 , piece_id=G7VZYID4TXGNPDN63NQULXOVSX42WMFFNPBX6KQJ5L2WSW4D2YGQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:14.831734493 (591h12m14.702817528s) [noise=true , address=185.167.204.206:4049 , piece_id=BQ7JEGSGPIFJUGZFBYRPDMBNPZ24BG6TEG4VUM4JUBTE4T2RF4SA, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:15.405688561 (591h12m15.276771606s) [noise=true , address=81.171.14.70:4027 , piece_id=6WEORWX5QOMFSJUEADIIBHHIPKNFP4NJPZAJADBUTPUSG7MZJMOQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:15.417110440 (591h12m15.288193485s) [noise=true , address=81.171.14.70:4031 , piece_id=U6Z4SNK32ZSWQ7NDR7ACXGRC4BADDOPR65ILBR64QLGVNDBADFFQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:15.425600446 (591h12m15.296683481s) [noise=true , address=81.171.14.70:4037 , piece_id=6A25LEIAQEJB3DNCTXQT6K7ZSKIGOW4QUGMNKLOG3CT2XXH25PSA, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:15.499056228 (591h12m15.370139263s) [noise=true , address=81.171.14.70:4038 , piece_id=3WRHW3VGXWHOEOV5Q6CLYSF7MUVWS7QHTNHLSTCKAUERPHAH3I3Q, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:15.514998925 (591h12m15.38608197s) [noise=true , address=81.171.14.70:4041 , piece_id=NORNA6WFTGPNOHU7MPTNNB5ONTBS5DJGCN2NWIZAEPYSUSIOSGEQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:15.528579544 (591h12m15.399662589s) [noise=true , address=81.171.14.70:4030 , piece_id=KITOJIT34CHS2QZBXCW2SJPZB4RHEHCMJ6IYQ4Z5KLL24YAWBIUQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:15.528991398 (591h12m15.400074434s) [noise=true , address=81.171.14.70:4047 , piece_id=ZWV5DP4YC3MSYDWEQH4ZK3VR4WZ4DUTI4VL3WVLQKFIEKIW44LOQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:15.534844544 (591h12m15.405927579s) [noise=true , address=81.171.14.70:4035 , piece_id=XCLL5SOAPJSSGF5YYQH6ZKGOHMLS6RI6H26FOW5XIJRY3TWYNEGA, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:15.915614910 (591h12m15.786697945s) [noise=true , address=81.171.14.70:4039 , piece_id=DEJDXFDHOSAMIP2FN5LJBJNBB4QLJSE2AUDW6IMA6IC4VRY7VAGQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:15.967820176 (591h12m15.838903211s) [noise=true , address=81.171.14.70:4046 , piece_id=B2MTUNKMUP5S7XW4SVQ5QI2ZO3VFB5QVAKBBY4BIZIUILM6O7VYQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:16.172116128 (591h12m16.043199163s) [noise=true , address=89.149.201.147:4007 , piece_id=HZLSYGSBP4TJH5V5XF3Q3AETJU625ERNDBD3VXW2HUZTTPAHLBMA, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:16.172137559 (591h12m16.043220604s) [seg_pos={0 0}, upload=518292]: Segment reached optimal threshold of 26 pieces.
Dec 30 06:41:16.172336949 (591h12m16.043419994s) [noise=true , address=89.149.201.147:4008 , piece_id=XYFRNJVIFFZA5S36KFPGZPVCPLRL2OW5QBJD3UR37VPPA2RX3RJA, seg_pos={0 0}, upload=518292]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QN8VP6o7qGw9Dd8EafU9xe7s7Cuf1FyqPMbnRaznrfPxfEUUn): protocol: expected piece hash; context canceled; EOF
Dec 30 06:41:16.172344213 (591h12m16.043427258s) [noise=true , address=89.149.201.147:4008 , piece_id=XYFRNJVIFFZA5S36KFPGZPVCPLRL2OW5QBJD3UR37VPPA2RX3RJA, seg_pos={0 0}, upload=518292]: Long tail cancellation due to optimal threshold
Dec 30 06:41:16.172554002 (591h12m16.043637037s) [noise=true , address=89.149.201.147:4020 , piece_id=N7G5DGEDYVKBTO5S6BYW3IYEOK7HAMK5JQIBMVPMPYRDHXODWEGA, seg_pos={0 0}, upload=518292]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12YeKHJH7CoouLFJYENJhzVLNkA32hJaqjLz3dXhr78eaSGtcic): protocol: expected piece hash; context canceled; EOF
Dec 30 06:41:16.172565233 (591h12m16.043648278s) [noise=true , address=89.149.201.147:4020 , piece_id=N7G5DGEDYVKBTO5S6BYW3IYEOK7HAMK5JQIBMVPMPYRDHXODWEGA, seg_pos={0 0}, upload=518292]: Long tail cancellation due to optimal threshold
Dec 30 06:41:16.172587506 (591h12m16.043670541s) [seg_pos={0 0}, upload=518292]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil>
Dec 30 06:41:16.172717945 (591h12m16.04380098s) [upload=518292]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 06:41:16.172724567 (591h12m16.043807602s) [upload=518292]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 06:41:16.404092237 (591h12m16.275175292s) [upload=518292]: Done uploading segments...
Dec 30 06:41:16.404119940 (591h12m16.275202975s) [upload=518292]: Upload finished. err: <nil>
Duration: 7.076617763s
core
uplink
eu-select-1
eu1-placement-15
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [...........X................................................] duration:
6.704396416s trace:
https://jaeger.dev.storj.io/trace/deb96436003dee
FAIL (2025-12-30 06:49:01 UTC)
orbiter warning: read tcp 188.166.225.170:52458->45.38.24.2:443: read: connection reset by peer
Duration: 6.704460288s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
4 out of 60 is failed (threshold=3) [..X...X......X............X.................................] duration:
15.923816288s trace:
https://jaeger.dev.storj.io/trace/45b769fc71e13448
FAIL (2025-12-30 06:48:42 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: context deadline exceeded
======LOGS======
.577697683 (590h53m11.448780728s) [noise=true , address=193.149.164.143:4704 , piece_id=4T2IJIA6WHI5LMIUP5BLJLCD3GRY3A6PXR6G5OWJWUSYZT5ELGHQ, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.578173528 (590h53m11.449256573s) [noise=true , address=64.49.13.27:4965 , piece_id=CO4O5F2IBZAAKBZBV3HWNNOXZOGLNNDPALWMQZ3DEM3MREASVFEA, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.578542992 (590h53m11.449626027s) [noise=true , address=77.90.31.15:4729 , piece_id=NVMXFEILHGT622IIPYYT4FFHC2WZLHMCZM4VT5WUPHRAPH3DX6TA, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.578846911 (590h53m11.449929946s) [noise=true , address=145.79.227.43:4177 , piece_id=YM37ZI3WNZ7SXGTFC224TOVVTHYNFIPRQB2TFLEZ735RNZE553FQ, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.579458044 (590h53m11.450541089s) [noise=true , address=45.91.116.11:4150 , piece_id=E66IDHTJDLYLYJIOWMOIKOOWLXCUNCMBIQ7SQBAD3VTJF6VX5JLQ, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.579891209 (590h53m11.450974245s) [noise=true , address=145.79.227.29:4945 , piece_id=SHL73IDEI54GRCWYNWHRUSJZZ4UUXREAX54DFJLTG5QPZCF5QKIQ, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.579915535 (590h53m11.45099858s) [noise=true , address=145.79.227.30:4772 , piece_id=PVMD5DWRRPVARTSZ6O7HN6MUVTZUYWLYSNQR5GDMNKUJZGXMYPFQ, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.580325466 (590h53m11.451408511s) [noise=true , address=77.90.31.12:4603 , piece_id=2I4L7F2WEGK5OSLODJSJ4672URBA7DUGXWZKTYB6ZNA2TQIGVXWA, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.580379599 (590h53m11.451462644s) [noise=true , address=77.90.31.14:4671 , piece_id=263TRISJ5ADFTX43ULSSGBAF2KMOXAPISJ56LLE53D74XOEMQ5EA, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.580692044 (590h53m11.451775089s) [noise=true , address=145.79.227.3:4175 , piece_id=UCV5WQNVWEAMEDBWDNXEVSQEGXPJE7KD6YSWC5A4WXYUKUCK4NLA, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.580857859 (590h53m11.451940894s) [noise=true , address=77.90.31.19:4860 , piece_id=U2MNFOAD7FHCDBE4FUYOROIP3P33ROJ7K75SF5Z747EFKFTF7RGA, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.581089210 (590h53m11.452172255s) [noise=true , address=145.79.227.44:4203 , piece_id=WHCKQ2FRLYAZE7Q6HYZNTS4EISVHZNAPDDKH34WN33ALK4MVK2EA, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.581351320 (590h53m11.452434355s) [noise=true , address=64.49.13.154:4928 , piece_id=SEVBZBMNGURP5TRUEBA4PHRSA6MNANVJUKINALJUWPQWBGKVYNTA, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.581457091 (590h53m11.452540126s) [noise=true , address=145.79.227.34:4909 , piece_id=3QSWTNR5LGUUAY3N5HJKSPL3TRQ6WO5F5OD4VTXKWHWUXUF7PVVQ, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.581863144 (590h53m11.452946179s) [noise=true , address=145.79.227.33:4881 , piece_id=EX66JF7U7A4ZSRWJMP4MCLQRPW2ZZFZFIUNV72XZXU7E5O3KWX3A, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.584010443 (590h53m11.455093478s) [noise=true , address=145.79.227.40:4070 , piece_id=YOAGDXNN4TTCK35LLQDSP24BOWLZXGNSQU7WMKEEVVMPXDVI74JA, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.585122791 (590h53m11.456205826s) [noise=true , address=45.91.116.14:4217 , piece_id=L4MAHTGKTNWBSN3YTB74E3DBJWVMA3DPU57MQK5FNK35DXRZ6AIA, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:12.983305402 (590h53m12.854388447s) [upload=517998]: Next returned nil segment
Dec 30 06:22:12.983321313 (590h53m12.854404348s) [upload=517998]: Waiting for error group managing segments...
Dec 30 06:22:13.379703053 (590h53m13.250786088s) [noise=true , address=64.49.13.27:4965 , piece_id=CO4O5F2IBZAAKBZBV3HWNNOXZOGLNNDPALWMQZ3DEM3MREASVFEA, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:13.418055622 (590h53m13.289138657s) [noise=true , address=145.79.227.3:4175 , piece_id=UCV5WQNVWEAMEDBWDNXEVSQEGXPJE7KD6YSWC5A4WXYUKUCK4NLA, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:13.509034160 (590h53m13.380117205s) [noise=true , address=64.49.13.154:4928 , piece_id=SEVBZBMNGURP5TRUEBA4PHRSA6MNANVJUKINALJUWPQWBGKVYNTA, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:13.642886154 (590h53m13.513969199s) [noise=true , address=77.90.31.14:4671 , piece_id=263TRISJ5ADFTX43ULSSGBAF2KMOXAPISJ56LLE53D74XOEMQ5EA, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:13.788145520 (590h53m13.659228555s) [noise=true , address=77.90.31.12:4603 , piece_id=2I4L7F2WEGK5OSLODJSJ4672URBA7DUGXWZKTYB6ZNA2TQIGVXWA, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:13.858878757 (590h53m13.729961792s) [noise=true , address=145.79.227.49:4369 , piece_id=XUEXGNJJIJ4DUBDNUV7CKF6J5JYHSLZAEPVGEDGBUBY42BIP4XEQ, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:13.879550569 (590h53m13.750633614s) [noise=true , address=145.79.227.34:4909 , piece_id=3QSWTNR5LGUUAY3N5HJKSPL3TRQ6WO5F5OD4VTXKWHWUXUF7PVVQ, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:13.940897723 (590h53m13.811980769s) [noise=true , address=77.90.31.19:4860 , piece_id=U2MNFOAD7FHCDBE4FUYOROIP3P33ROJ7K75SF5Z747EFKFTF7RGA, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:13.989791936 (590h53m13.860874981s) [noise=true , address=45.91.116.11:4150 , piece_id=E66IDHTJDLYLYJIOWMOIKOOWLXCUNCMBIQ7SQBAD3VTJF6VX5JLQ, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:14.289516718 (590h53m14.160599753s) [noise=true , address=145.79.227.44:4203 , piece_id=WHCKQ2FRLYAZE7Q6HYZNTS4EISVHZNAPDDKH34WN33ALK4MVK2EA, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:14.338651537 (590h53m14.209734582s) [noise=true , address=77.90.31.15:4729 , piece_id=NVMXFEILHGT622IIPYYT4FFHC2WZLHMCZM4VT5WUPHRAPH3DX6TA, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:14.344145168 (590h53m14.215228213s) [noise=true , address=145.79.227.30:4772 , piece_id=PVMD5DWRRPVARTSZ6O7HN6MUVTZUYWLYSNQR5GDMNKUJZGXMYPFQ, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:14.667232847 (590h53m14.538315892s) [noise=true , address=193.149.164.143:4704 , piece_id=4T2IJIA6WHI5LMIUP5BLJLCD3GRY3A6PXR6G5OWJWUSYZT5ELGHQ, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:14.671388990 (590h53m14.542472025s) [noise=true , address=145.79.227.29:4945 , piece_id=SHL73IDEI54GRCWYNWHRUSJZZ4UUXREAX54DFJLTG5QPZCF5QKIQ, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:14.671415190 (590h53m14.542498226s) [seg_pos={0 0}, upload=517998]: Segment reached optimal threshold of 14 pieces.
Dec 30 06:22:14.671803600 (590h53m14.542886655s) [noise=true , address=145.79.227.40:4070 , piece_id=YOAGDXNN4TTCK35LLQDSP24BOWLZXGNSQU7WMKEEVVMPXDVI74JA, seg_pos={0 0}, upload=517998]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12hU1LwL2gX7rVnrNgzaBCYB37ziqf7s7aUgtF2Uc3antReQ6me): protocol: expected piece hash; context canceled; EOF
Dec 30 06:22:14.671814470 (590h53m14.542897525s) [noise=true , address=145.79.227.40:4070 , piece_id=YOAGDXNN4TTCK35LLQDSP24BOWLZXGNSQU7WMKEEVVMPXDVI74JA, seg_pos={0 0}, upload=517998]: Long tail cancellation due to optimal threshold
Dec 30 06:22:14.672014932 (590h53m14.543097977s) [noise=true , address=145.79.227.43:4177 , piece_id=YM37ZI3WNZ7SXGTFC224TOVVTHYNFIPRQB2TFLEZ735RNZE553FQ, seg_pos={0 0}, upload=517998]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bhN3cR22QuAYihnnB3x3R97UZRWJ8sFPP8fjQcXZ7qhNZphZG): protocol: expected piece hash; context canceled; EOF
Dec 30 06:22:14.672022637 (590h53m14.543105672s) [noise=true , address=145.79.227.43:4177 , piece_id=YM37ZI3WNZ7SXGTFC224TOVVTHYNFIPRQB2TFLEZ735RNZE553FQ, seg_pos={0 0}, upload=517998]: Long tail cancellation due to optimal threshold
Dec 30 06:22:14.672088933 (590h53m14.543171988s) [noise=true , address=145.79.227.33:4881 , piece_id=EX66JF7U7A4ZSRWJMP4MCLQRPW2ZZFZFIUNV72XZXU7E5O3KWX3A, seg_pos={0 0}, upload=517998]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mMBjg1b4MqTZPE5NpNNMYDzmsHqWYotE5jBNmZLyA5FyYLZRp): protocol: expected piece hash; context canceled; EOF
Dec 30 06:22:14.672127677 (590h53m14.543210742s) [noise=true , address=145.79.227.33:4881 , piece_id=EX66JF7U7A4ZSRWJMP4MCLQRPW2ZZFZFIUNV72XZXU7E5O3KWX3A, seg_pos={0 0}, upload=517998]: Long tail cancellation due to optimal threshold
Dec 30 06:22:14.672312739 (590h53m14.543395774s) [noise=true , address=45.91.116.14:4217 , piece_id=L4MAHTGKTNWBSN3YTB74E3DBJWVMA3DPU57MQK5FNK35DXRZ6AIA, seg_pos={0 0}, upload=517998]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xSXWFiq27Dd2vBDD9vVnhDHVG59UmDKn3hVAhf3oyo5goVNaE): protocol: expected piece hash; context canceled; EOF
Dec 30 06:22:14.672319953 (590h53m14.543402988s) [noise=true , address=45.91.116.14:4217 , piece_id=L4MAHTGKTNWBSN3YTB74E3DBJWVMA3DPU57MQK5FNK35DXRZ6AIA, seg_pos={0 0}, upload=517998]: Long tail cancellation due to optimal threshold
Dec 30 06:22:14.672337847 (590h53m14.543420882s) [seg_pos={0 0}, upload=517998]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 30 06:22:14.672452947 (590h53m14.543535982s) [upload=517998]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 06:22:14.672458507 (590h53m14.543541542s) [upload=517998]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 06:22:14.981437069 (590h53m14.852520114s) [upload=517998]: Done uploading segments...
Dec 30 06:22:14.981456546 (590h53m14.852539581s) [upload=517998]: Upload finished. err: <nil>
Duration: 15.923830474s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [............X...............................................] duration:
7.01877124s trace:
https://jaeger.dev.storj.io/trace/deb96436003dee
FAIL (2025-12-30 06:48:42 UTC)
orbiter warning: read tcp 188.166.225.170:52458->45.38.24.2:443: read: connection reset by peer
Duration: 7.018824962s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
14 out of 60 is failed (threshold=3) [..X......X....X...X..X.X.X.....X.......X.X.....X....XX...X..] duration:
5.909713581s trace:
https://jaeger.dev.storj.io/trace/25efa104cb3576c8
FAIL (2025-12-30 06:48:39 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW, Piece ID: DG5N5IK35GLLLBJKWK7J53N3ER737OAGQBCHQ53AVMTIJHNWRXEA) context deadline exceeded; piecestore: (Node ID: 1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w, Piece ID: TSCFP4MGODAU4IOWHIKWBKNNAE4G2ZGCBTVRKF3KVXCCP7Z6H7EQ) context deadline exceeded
======LOGS======
30 06:47:33.902700925 (591h18m33.77378396s) [upload=518373]: Starting upload
Dec 30 06:47:33.902775336 (591h18m33.773858372s) [upload=518373]: Done starting upload
Dec 30 06:47:33.902944639 (591h18m33.774027674s) [upload=518373]: Uploading segments...
Dec 30 06:47:33.903006506 (591h18m33.774089541s) [upload=518373]: Got next segment. Inline: false
Dec 30 06:47:33.903036744 (591h18m33.774119779s) [upload=518373]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 30 06:47:33.903043197 (591h18m33.774126232s) [upload=518373]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 30 06:47:34.148006034 (591h18m34.019089069s) [seg_pos={0 0}, upload=518373]: Begin upload segment...
Dec 30 06:47:34.148062040 (591h18m34.019145075s) [seg_pos={0 0}, upload=518373]: Done begin upload segment.
Dec 30 06:47:34.148107336 (591h18m34.019190371s) [noise=true , address=45.91.116.5:4297 , piece_id=DG5N5IK35GLLLBJKWK7J53N3ER737OAGQBCHQ53AVMTIJHNWRXEA, seg_pos={0 0}, upload=518373]: Uploading piece...
Dec 30 06:47:34.148155579 (591h18m34.019238624s) [noise=true , address=145.79.227.46:4603 , piece_id=535JSH3VHKZETEICNVLFU3VCROFZVTZGOZRTN4O3UEIUDIQ6QEIQ, seg_pos={0 0}, upload=518373]: Uploading piece...
Dec 30 06:47:34.148683915 (591h18m34.01976695s) [noise=true , address=145.79.227.40:4051 , piece_id=N7JIQ4XFYDM4XQC4SMCUF2VYGGY35VA5BOKVCNAKD7BSKUSAPVNQ, seg_pos={0 0}, upload=518373]: Uploading piece...
Dec 30 06:47:34.148742016 (591h18m34.019825051s) [noise=true , address=145.79.227.48:4301 , piece_id=WMF7RPBJ3HUKXUJ63RLZFI6MVLUBJI3NLE4V7FS7NIG54XU5KLDA, seg_pos={0 0}, upload=518373]: Uploading piece...
Dec 30 06:47:34.149083746 (591h18m34.020166781s) [noise=true , address=45.91.116.4:4939 , piece_id=DRYXDGA2FTS2T5F47ZXZASU6O63V6LVCAVZPKK3OJTO5Y4IERHUQ, seg_pos={0 0}, upload=518373]: Uploading piece...
Dec 30 06:47:34.149391432 (591h18m34.020474477s) [noise=true , address=45.91.116.7:4010 , piece_id=TSCFP4MGODAU4IOWHIKWBKNNAE4G2ZGCBTVRKF3KVXCCP7Z6H7EQ, seg_pos={0 0}, upload=518373]: Uploading piece...
Dec 30 06:47:34.149687105 (591h18m34.02077014s) [noise=true , address=145.79.227.47:4265 , piece_id=FAVM24T5F2HBQIQS7IQ3GNVA5FNBCPZAUTJDVL5K4QYEGDI67MZQ, seg_pos={0 0}, upload=518373]: Uploading piece...
Dec 30 06:47:34.149987156 (591h18m34.021070191s) [noise=true , address=145.79.227.41:4085 , piece_id=V7X4XLDOB6JATX4JOWZ72XC7ONEDN64FIBBTZLJN2JLURNZRSH6Q, seg_pos={0 0}, upload=518373]: Uploading piece...
Dec 30 06:47:34.150321904 (591h18m34.021404949s) [noise=true , address=45.91.116.11:4120 , piece_id=CGPZDXJ44PHSBJ5NJUNKEUEFNYHUJPVWUETQGQ5HWAW3YZW4QDVA, seg_pos={0 0}, upload=518373]: Uploading piece...
Dec 30 06:47:34.150333786 (591h18m34.021416821s) [noise=true , address=145.79.227.39:4014 , piece_id=BLNHJN77DG2Y34Y3Z663ZUMQTBBOHWZKSW3QMBGLCVLNL6CEGJOA, seg_pos={0 0}, upload=518373]: Uploading piece...
Dec 30 06:47:34.150579524 (591h18m34.021662559s) [noise=true , address=145.79.227.19:4078 , piece_id=C2VTC5RDJBTU5Z2KKJENXLKYY2WIWBFELGDCRTJLRGSS3HWCG3XQ, seg_pos={0 0}, upload=518373]: Uploading piece...
Dec 30 06:47:34.150705945 (591h18m34.02178898s) [noise=true , address=145.79.227.32:4834 , piece_id=BQL2WD5B6ZKNH7B4AX7ELFSMKQVLVJCELGIDJYPKVSHURFJSRJLQ, seg_pos={0 0}, upload=518373]: Uploading piece...
Dec 30 06:47:34.151198954 (591h18m34.022281989s) [noise=true , address=45.91.116.14:4191 , piece_id=S6RREAK6WN5LM7T7TKBMPTDB7G2UWNGZ2L244ULRIMBEA6MOWGLA, seg_pos={0 0}, upload=518373]: Uploading piece...
Dec 30 06:47:34.151603264 (591h18m34.022686299s) [noise=true , address=145.79.227.43:4159 , piece_id=G55LMZZF32WFDATSY2BQKII65LYTA3IXEW5FIKYV444SNWQ7R2SQ, seg_pos={0 0}, upload=518373]: Uploading piece...
Dec 30 06:47:34.151984440 (591h18m34.023067485s) [noise=true , address=145.79.227.2:4114 , piece_id=QUHSCSX6TMOUGUKSUMAOENS4UBW7W4TBCTY4GAOCDH27IK2EQRBQ, seg_pos={0 0}, upload=518373]: Uploading piece...
Dec 30 06:47:34.152395842 (591h18m34.023478877s) [noise=true , address=145.79.227.17:4970 , piece_id=OR5LHPTUBOS3DDI4W2XFT5ED3Y5FGDOOVP6VPOHYYLELNS7SZB3A, seg_pos={0 0}, upload=518373]: Uploading piece...
Dec 30 06:47:35.844170154 (591h18m35.715253189s) [upload=518373]: Next returned nil segment
Dec 30 06:47:35.844218737 (591h18m35.715301792s) [upload=518373]: Waiting for error group managing segments...
Dec 30 06:47:36.127146455 (591h18m35.9982295s) [noise=true , address=145.79.227.19:4078 , piece_id=C2VTC5RDJBTU5Z2KKJENXLKYY2WIWBFELGDCRTJLRGSS3HWCG3XQ, seg_pos={0 0}, upload=518373]: Done uploading piece. err: <nil>
Dec 30 06:47:36.130047630 (591h18m36.001130665s) [noise=true , address=145.79.227.39:4014 , piece_id=BLNHJN77DG2Y34Y3Z663ZUMQTBBOHWZKSW3QMBGLCVLNL6CEGJOA, seg_pos={0 0}, upload=518373]: Done uploading piece. err: <nil>
Dec 30 06:47:36.140822898 (591h18m36.011905933s) [noise=true , address=145.79.227.2:4114 , piece_id=QUHSCSX6TMOUGUKSUMAOENS4UBW7W4TBCTY4GAOCDH27IK2EQRBQ, seg_pos={0 0}, upload=518373]: Done uploading piece. err: <nil>
Dec 30 06:47:36.141865904 (591h18m36.012948949s) [noise=true , address=145.79.227.46:4603 , piece_id=535JSH3VHKZETEICNVLFU3VCROFZVTZGOZRTN4O3UEIUDIQ6QEIQ, seg_pos={0 0}, upload=518373]: Done uploading piece. err: <nil>
Dec 30 06:47:36.142992949 (591h18m36.014075984s) [noise=true , address=145.79.227.48:4301 , piece_id=WMF7RPBJ3HUKXUJ63RLZFI6MVLUBJI3NLE4V7FS7NIG54XU5KLDA, seg_pos={0 0}, upload=518373]: Done uploading piece. err: <nil>
Dec 30 06:47:36.154997809 (591h18m36.026080844s) [noise=true , address=145.79.227.43:4159 , piece_id=G55LMZZF32WFDATSY2BQKII65LYTA3IXEW5FIKYV444SNWQ7R2SQ, seg_pos={0 0}, upload=518373]: Done uploading piece. err: <nil>
Dec 30 06:47:36.167251051 (591h18m36.038334096s) [noise=true , address=145.79.227.41:4085 , piece_id=V7X4XLDOB6JATX4JOWZ72XC7ONEDN64FIBBTZLJN2JLURNZRSH6Q, seg_pos={0 0}, upload=518373]: Done uploading piece. err: <nil>
Dec 30 06:47:36.171358592 (591h18m36.042441637s) [noise=true , address=145.79.227.47:4265 , piece_id=FAVM24T5F2HBQIQS7IQ3GNVA5FNBCPZAUTJDVL5K4QYEGDI67MZQ, seg_pos={0 0}, upload=518373]: Done uploading piece. err: <nil>
Dec 30 06:47:36.204427992 (591h18m36.075511027s) [noise=true , address=145.79.227.32:4834 , piece_id=BQL2WD5B6ZKNH7B4AX7ELFSMKQVLVJCELGIDJYPKVSHURFJSRJLQ, seg_pos={0 0}, upload=518373]: Done uploading piece. err: <nil>
Dec 30 06:47:36.219970897 (591h18m36.091053932s) [noise=true , address=145.79.227.17:4970 , piece_id=OR5LHPTUBOS3DDI4W2XFT5ED3Y5FGDOOVP6VPOHYYLELNS7SZB3A, seg_pos={0 0}, upload=518373]: Done uploading piece. err: <nil>
Dec 30 06:47:36.460845771 (591h18m36.331928816s) [noise=true , address=45.91.116.4:4939 , piece_id=DRYXDGA2FTS2T5F47ZXZASU6O63V6LVCAVZPKK3OJTO5Y4IERHUQ, seg_pos={0 0}, upload=518373]: Done uploading piece. err: <nil>
Dec 30 06:47:36.559892892 (591h18m36.430975927s) [noise=true , address=45.91.116.5:4297 , piece_id=DG5N5IK35GLLLBJKWK7J53N3ER737OAGQBCHQ53AVMTIJHNWRXEA, seg_pos={0 0}, upload=518373]: Done uploading piece. err: <nil>
Dec 30 06:47:36.564095394 (591h18m36.435178429s) [noise=true , address=45.91.116.7:4010 , piece_id=TSCFP4MGODAU4IOWHIKWBKNNAE4G2ZGCBTVRKF3KVXCCP7Z6H7EQ, seg_pos={0 0}, upload=518373]: Done uploading piece. err: <nil>
Dec 30 06:47:36.564160248 (591h18m36.435243283s) [seg_pos={0 0}, upload=518373]: Segment reached optimal threshold of 13 pieces.
Dec 30 06:47:36.564563686 (591h18m36.435646731s) [noise=true , address=45.91.116.11:4120 , piece_id=CGPZDXJ44PHSBJ5NJUNKEUEFNYHUJPVWUETQGQ5HWAW3YZW4QDVA, seg_pos={0 0}, upload=518373]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8): protocol: write tcp 188.166.225.170:50072->45.91.116.11:4120: use of closed network connection; write tcp 188.166.225.170:50072->45.91.116.11:4120: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50072->45.91.116.11:4120: use of closed network connection
Dec 30 06:47:36.564574417 (591h18m36.435657452s) [noise=true , address=45.91.116.11:4120 , piece_id=CGPZDXJ44PHSBJ5NJUNKEUEFNYHUJPVWUETQGQ5HWAW3YZW4QDVA, seg_pos={0 0}, upload=518373]: Long tail cancellation due to optimal threshold
Dec 30 06:47:36.564877924 (591h18m36.435960959s) [noise=true , address=45.91.116.14:4191 , piece_id=S6RREAK6WN5LM7T7TKBMPTDB7G2UWNGZ2L244ULRIMBEA6MOWGLA, seg_pos={0 0}, upload=518373]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq): protocol: expected piece hash; context canceled; EOF
Dec 30 06:47:36.564886651 (591h18m36.435969696s) [noise=true , address=45.91.116.14:4191 , piece_id=S6RREAK6WN5LM7T7TKBMPTDB7G2UWNGZ2L244ULRIMBEA6MOWGLA, seg_pos={0 0}, upload=518373]: Long tail cancellation due to optimal threshold
Dec 30 06:47:36.565006689 (591h18m36.436089734s) [noise=true , address=145.79.227.40:4051 , piece_id=N7JIQ4XFYDM4XQC4SMCUF2VYGGY35VA5BOKVCNAKD7BSKUSAPVNQ, seg_pos={0 0}, upload=518373]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rPbTARUrq7YsD8X75ocnyEvC517itk79s3Dp1wUg72fVuBS8n): protocol: expected piece hash; context canceled; EOF
Dec 30 06:47:36.565014714 (591h18m36.436097749s) [noise=true , address=145.79.227.40:4051 , piece_id=N7JIQ4XFYDM4XQC4SMCUF2VYGGY35VA5BOKVCNAKD7BSKUSAPVNQ, seg_pos={0 0}, upload=518373]: Long tail cancellation due to optimal threshold
Dec 30 06:47:36.565048769 (591h18m36.436131804s) [seg_pos={0 0}, upload=518373]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 30 06:47:36.565134443 (591h18m36.436217478s) [upload=518373]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 06:47:36.565140214 (591h18m36.436223259s) [upload=518373]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 06:47:36.856746186 (591h18m36.727829231s) [upload=518373]: Done uploading segments...
Dec 30 06:47:36.856766275 (591h18m36.72784932s) [upload=518373]: Upload finished. err: <nil>
Duration: 5.909740511s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
4 out of 60 is failed (threshold=3) [...X...X......X............X................................] duration:
37.791232085s trace:
https://jaeger.dev.storj.io/trace/45b769fc71e13448
FAIL (2025-12-30 06:48:26 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: context deadline exceeded
======LOGS======
.577697683 (590h53m11.448780728s) [noise=true , address=193.149.164.143:4704 , piece_id=4T2IJIA6WHI5LMIUP5BLJLCD3GRY3A6PXR6G5OWJWUSYZT5ELGHQ, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.578173528 (590h53m11.449256573s) [noise=true , address=64.49.13.27:4965 , piece_id=CO4O5F2IBZAAKBZBV3HWNNOXZOGLNNDPALWMQZ3DEM3MREASVFEA, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.578542992 (590h53m11.449626027s) [noise=true , address=77.90.31.15:4729 , piece_id=NVMXFEILHGT622IIPYYT4FFHC2WZLHMCZM4VT5WUPHRAPH3DX6TA, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.578846911 (590h53m11.449929946s) [noise=true , address=145.79.227.43:4177 , piece_id=YM37ZI3WNZ7SXGTFC224TOVVTHYNFIPRQB2TFLEZ735RNZE553FQ, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.579458044 (590h53m11.450541089s) [noise=true , address=45.91.116.11:4150 , piece_id=E66IDHTJDLYLYJIOWMOIKOOWLXCUNCMBIQ7SQBAD3VTJF6VX5JLQ, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.579891209 (590h53m11.450974245s) [noise=true , address=145.79.227.29:4945 , piece_id=SHL73IDEI54GRCWYNWHRUSJZZ4UUXREAX54DFJLTG5QPZCF5QKIQ, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.579915535 (590h53m11.45099858s) [noise=true , address=145.79.227.30:4772 , piece_id=PVMD5DWRRPVARTSZ6O7HN6MUVTZUYWLYSNQR5GDMNKUJZGXMYPFQ, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.580325466 (590h53m11.451408511s) [noise=true , address=77.90.31.12:4603 , piece_id=2I4L7F2WEGK5OSLODJSJ4672URBA7DUGXWZKTYB6ZNA2TQIGVXWA, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.580379599 (590h53m11.451462644s) [noise=true , address=77.90.31.14:4671 , piece_id=263TRISJ5ADFTX43ULSSGBAF2KMOXAPISJ56LLE53D74XOEMQ5EA, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.580692044 (590h53m11.451775089s) [noise=true , address=145.79.227.3:4175 , piece_id=UCV5WQNVWEAMEDBWDNXEVSQEGXPJE7KD6YSWC5A4WXYUKUCK4NLA, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.580857859 (590h53m11.451940894s) [noise=true , address=77.90.31.19:4860 , piece_id=U2MNFOAD7FHCDBE4FUYOROIP3P33ROJ7K75SF5Z747EFKFTF7RGA, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.581089210 (590h53m11.452172255s) [noise=true , address=145.79.227.44:4203 , piece_id=WHCKQ2FRLYAZE7Q6HYZNTS4EISVHZNAPDDKH34WN33ALK4MVK2EA, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.581351320 (590h53m11.452434355s) [noise=true , address=64.49.13.154:4928 , piece_id=SEVBZBMNGURP5TRUEBA4PHRSA6MNANVJUKINALJUWPQWBGKVYNTA, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.581457091 (590h53m11.452540126s) [noise=true , address=145.79.227.34:4909 , piece_id=3QSWTNR5LGUUAY3N5HJKSPL3TRQ6WO5F5OD4VTXKWHWUXUF7PVVQ, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.581863144 (590h53m11.452946179s) [noise=true , address=145.79.227.33:4881 , piece_id=EX66JF7U7A4ZSRWJMP4MCLQRPW2ZZFZFIUNV72XZXU7E5O3KWX3A, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.584010443 (590h53m11.455093478s) [noise=true , address=145.79.227.40:4070 , piece_id=YOAGDXNN4TTCK35LLQDSP24BOWLZXGNSQU7WMKEEVVMPXDVI74JA, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.585122791 (590h53m11.456205826s) [noise=true , address=45.91.116.14:4217 , piece_id=L4MAHTGKTNWBSN3YTB74E3DBJWVMA3DPU57MQK5FNK35DXRZ6AIA, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:12.983305402 (590h53m12.854388447s) [upload=517998]: Next returned nil segment
Dec 30 06:22:12.983321313 (590h53m12.854404348s) [upload=517998]: Waiting for error group managing segments...
Dec 30 06:22:13.379703053 (590h53m13.250786088s) [noise=true , address=64.49.13.27:4965 , piece_id=CO4O5F2IBZAAKBZBV3HWNNOXZOGLNNDPALWMQZ3DEM3MREASVFEA, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:13.418055622 (590h53m13.289138657s) [noise=true , address=145.79.227.3:4175 , piece_id=UCV5WQNVWEAMEDBWDNXEVSQEGXPJE7KD6YSWC5A4WXYUKUCK4NLA, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:13.509034160 (590h53m13.380117205s) [noise=true , address=64.49.13.154:4928 , piece_id=SEVBZBMNGURP5TRUEBA4PHRSA6MNANVJUKINALJUWPQWBGKVYNTA, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:13.642886154 (590h53m13.513969199s) [noise=true , address=77.90.31.14:4671 , piece_id=263TRISJ5ADFTX43ULSSGBAF2KMOXAPISJ56LLE53D74XOEMQ5EA, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:13.788145520 (590h53m13.659228555s) [noise=true , address=77.90.31.12:4603 , piece_id=2I4L7F2WEGK5OSLODJSJ4672URBA7DUGXWZKTYB6ZNA2TQIGVXWA, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:13.858878757 (590h53m13.729961792s) [noise=true , address=145.79.227.49:4369 , piece_id=XUEXGNJJIJ4DUBDNUV7CKF6J5JYHSLZAEPVGEDGBUBY42BIP4XEQ, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:13.879550569 (590h53m13.750633614s) [noise=true , address=145.79.227.34:4909 , piece_id=3QSWTNR5LGUUAY3N5HJKSPL3TRQ6WO5F5OD4VTXKWHWUXUF7PVVQ, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:13.940897723 (590h53m13.811980769s) [noise=true , address=77.90.31.19:4860 , piece_id=U2MNFOAD7FHCDBE4FUYOROIP3P33ROJ7K75SF5Z747EFKFTF7RGA, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:13.989791936 (590h53m13.860874981s) [noise=true , address=45.91.116.11:4150 , piece_id=E66IDHTJDLYLYJIOWMOIKOOWLXCUNCMBIQ7SQBAD3VTJF6VX5JLQ, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:14.289516718 (590h53m14.160599753s) [noise=true , address=145.79.227.44:4203 , piece_id=WHCKQ2FRLYAZE7Q6HYZNTS4EISVHZNAPDDKH34WN33ALK4MVK2EA, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:14.338651537 (590h53m14.209734582s) [noise=true , address=77.90.31.15:4729 , piece_id=NVMXFEILHGT622IIPYYT4FFHC2WZLHMCZM4VT5WUPHRAPH3DX6TA, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:14.344145168 (590h53m14.215228213s) [noise=true , address=145.79.227.30:4772 , piece_id=PVMD5DWRRPVARTSZ6O7HN6MUVTZUYWLYSNQR5GDMNKUJZGXMYPFQ, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:14.667232847 (590h53m14.538315892s) [noise=true , address=193.149.164.143:4704 , piece_id=4T2IJIA6WHI5LMIUP5BLJLCD3GRY3A6PXR6G5OWJWUSYZT5ELGHQ, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:14.671388990 (590h53m14.542472025s) [noise=true , address=145.79.227.29:4945 , piece_id=SHL73IDEI54GRCWYNWHRUSJZZ4UUXREAX54DFJLTG5QPZCF5QKIQ, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:14.671415190 (590h53m14.542498226s) [seg_pos={0 0}, upload=517998]: Segment reached optimal threshold of 14 pieces.
Dec 30 06:22:14.671803600 (590h53m14.542886655s) [noise=true , address=145.79.227.40:4070 , piece_id=YOAGDXNN4TTCK35LLQDSP24BOWLZXGNSQU7WMKEEVVMPXDVI74JA, seg_pos={0 0}, upload=517998]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12hU1LwL2gX7rVnrNgzaBCYB37ziqf7s7aUgtF2Uc3antReQ6me): protocol: expected piece hash; context canceled; EOF
Dec 30 06:22:14.671814470 (590h53m14.542897525s) [noise=true , address=145.79.227.40:4070 , piece_id=YOAGDXNN4TTCK35LLQDSP24BOWLZXGNSQU7WMKEEVVMPXDVI74JA, seg_pos={0 0}, upload=517998]: Long tail cancellation due to optimal threshold
Dec 30 06:22:14.672014932 (590h53m14.543097977s) [noise=true , address=145.79.227.43:4177 , piece_id=YM37ZI3WNZ7SXGTFC224TOVVTHYNFIPRQB2TFLEZ735RNZE553FQ, seg_pos={0 0}, upload=517998]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bhN3cR22QuAYihnnB3x3R97UZRWJ8sFPP8fjQcXZ7qhNZphZG): protocol: expected piece hash; context canceled; EOF
Dec 30 06:22:14.672022637 (590h53m14.543105672s) [noise=true , address=145.79.227.43:4177 , piece_id=YM37ZI3WNZ7SXGTFC224TOVVTHYNFIPRQB2TFLEZ735RNZE553FQ, seg_pos={0 0}, upload=517998]: Long tail cancellation due to optimal threshold
Dec 30 06:22:14.672088933 (590h53m14.543171988s) [noise=true , address=145.79.227.33:4881 , piece_id=EX66JF7U7A4ZSRWJMP4MCLQRPW2ZZFZFIUNV72XZXU7E5O3KWX3A, seg_pos={0 0}, upload=517998]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mMBjg1b4MqTZPE5NpNNMYDzmsHqWYotE5jBNmZLyA5FyYLZRp): protocol: expected piece hash; context canceled; EOF
Dec 30 06:22:14.672127677 (590h53m14.543210742s) [noise=true , address=145.79.227.33:4881 , piece_id=EX66JF7U7A4ZSRWJMP4MCLQRPW2ZZFZFIUNV72XZXU7E5O3KWX3A, seg_pos={0 0}, upload=517998]: Long tail cancellation due to optimal threshold
Dec 30 06:22:14.672312739 (590h53m14.543395774s) [noise=true , address=45.91.116.14:4217 , piece_id=L4MAHTGKTNWBSN3YTB74E3DBJWVMA3DPU57MQK5FNK35DXRZ6AIA, seg_pos={0 0}, upload=517998]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xSXWFiq27Dd2vBDD9vVnhDHVG59UmDKn3hVAhf3oyo5goVNaE): protocol: expected piece hash; context canceled; EOF
Dec 30 06:22:14.672319953 (590h53m14.543402988s) [noise=true , address=45.91.116.14:4217 , piece_id=L4MAHTGKTNWBSN3YTB74E3DBJWVMA3DPU57MQK5FNK35DXRZ6AIA, seg_pos={0 0}, upload=517998]: Long tail cancellation due to optimal threshold
Dec 30 06:22:14.672337847 (590h53m14.543420882s) [seg_pos={0 0}, upload=517998]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 30 06:22:14.672452947 (590h53m14.543535982s) [upload=517998]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 06:22:14.672458507 (590h53m14.543541542s) [upload=517998]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 06:22:14.981437069 (590h53m14.852520114s) [upload=517998]: Done uploading segments...
Dec 30 06:22:14.981456546 (590h53m14.852539581s) [upload=517998]: Upload finished. err: <nil>
Duration: 37.791293341s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
10 out of 60 is failed (threshold=6) [.X...........X..............XX...X......X..X..X.....X.....X.] duration:
3.681720498s trace:
https://jaeger.dev.storj.io/trace/2dfc5fc3a860afad
FAIL (2025-12-30 06:48:25 UTC)
orbiter warning: context deadline exceeded
Duration: 3.681748731s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)antiblip:
1 out of 60 is failed (threshold=3) [..................................................X.........] duration:
9.284869566s trace:
https://jaeger.dev.storj.io/trace/4c2f4efaff0e0ea0
FAIL (2025-12-30 06:48:13 UTC)
orbiter warning:
operation: DeleteObject
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
load=518292]: Uploading piece...
Dec 30 06:41:13.598100152 (591h12m13.469183187s) [noise=true , address=81.171.14.70:4041 , piece_id=NORNA6WFTGPNOHU7MPTNNB5ONTBS5DJGCN2NWIZAEPYSUSIOSGEQ, seg_pos={0 0}, upload=518292]: Uploading piece...
Dec 30 06:41:13.598587971 (591h12m13.469671016s) [noise=true , address=89.149.201.147:4007 , piece_id=HZLSYGSBP4TJH5V5XF3Q3AETJU625ERNDBD3VXW2HUZTTPAHLBMA, seg_pos={0 0}, upload=518292]: Uploading piece...
Dec 30 06:41:13.599093143 (591h12m13.470176178s) [noise=true , address=89.149.201.147:4008 , piece_id=XYFRNJVIFFZA5S36KFPGZPVCPLRL2OW5QBJD3UR37VPPA2RX3RJA, seg_pos={0 0}, upload=518292]: Uploading piece...
Dec 30 06:41:13.599699999 (591h12m13.470783034s) [noise=true , address=81.171.14.70:4039 , piece_id=DEJDXFDHOSAMIP2FN5LJBJNBB4QLJSE2AUDW6IMA6IC4VRY7VAGQ, seg_pos={0 0}, upload=518292]: Uploading piece...
Dec 30 06:41:13.600136169 (591h12m13.471219204s) [noise=true , address=185.167.204.205:4034 , piece_id=OWZHQB5ZDT2QZ2QAHZJVRWQ222MX6CWLKJUOIAAGSMAVLWGJAOBA, seg_pos={0 0}, upload=518292]: Uploading piece...
Dec 30 06:41:13.600543184 (591h12m13.471626229s) [noise=true , address=185.167.204.206:4058 , piece_id=ROOCGK64M5RHIWNAVCSZNYTZ6M2ORWB7MJ27XJVDTFVQIKJZHY4A, seg_pos={0 0}, upload=518292]: Uploading piece...
Dec 30 06:41:13.601048286 (591h12m13.472131332s) [noise=true , address=81.171.14.70:4046 , piece_id=B2MTUNKMUP5S7XW4SVQ5QI2ZO3VFB5QVAKBBY4BIZIUILM6O7VYQ, seg_pos={0 0}, upload=518292]: Uploading piece...
Dec 30 06:41:13.601698784 (591h12m13.472781819s) [noise=true , address=185.167.204.206:4064 , piece_id=H4PP7HPHXRX2LOZ4776AMWAJDN7OC435E2K4BXI7XOZYKTB4YUNA, seg_pos={0 0}, upload=518292]: Uploading piece...
Dec 30 06:41:13.602168259 (591h12m13.473251304s) [noise=true , address=185.167.204.206:4065 , piece_id=LD3J4656Q5QEUCMJ5OKEYATEVRIKSXO7XGARTN25BVM7FDIYOWKA, seg_pos={0 0}, upload=518292]: Uploading piece...
Dec 30 06:41:13.602782708 (591h12m13.473865753s) [noise=true , address=81.171.14.70:4027 , piece_id=6WEORWX5QOMFSJUEADIIBHHIPKNFP4NJPZAJADBUTPUSG7MZJMOQ, seg_pos={0 0}, upload=518292]: Uploading piece...
Dec 30 06:41:14.423395002 (591h12m14.294478047s) [upload=518292]: Next returned nil segment
Dec 30 06:41:14.423414068 (591h12m14.294497103s) [upload=518292]: Waiting for error group managing segments...
Dec 30 06:41:14.817650316 (591h12m14.688733351s) [noise=true , address=185.167.204.205:4029 , piece_id=KUOMMYJ2F4NIK6VN4C5M3Y3TMHYEZH7QFNJ6K3CLOAVI5R6VKTHQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:14.817870054 (591h12m14.688953099s) [noise=true , address=185.167.204.205:4019 , piece_id=H6AH654VGX6NHBUKODIWGL7I2RKE3DDMY4UM6VDONP2EP3NPDJBA, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:14.818623698 (591h12m14.689706744s) [noise=true , address=185.167.204.206:4065 , piece_id=LD3J4656Q5QEUCMJ5OKEYATEVRIKSXO7XGARTN25BVM7FDIYOWKA, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:14.818932436 (591h12m14.690015481s) [noise=true , address=185.167.204.206:4061 , piece_id=T2XS4IFPQSVS3T45SPQ5ZDDVT2VMTMXHKL2ZMX4OIOPRR67SCD2Q, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:14.819147046 (591h12m14.690230091s) [noise=true , address=185.167.204.206:4071 , piece_id=QI2TOY5KCDZVWCCLMAJXVUWD2HB7S7IOTEAV2EHAMUNHKOFKTNYA, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:14.819678578 (591h12m14.690761623s) [noise=true , address=185.167.204.206:4059 , piece_id=LMOGTWX2YIRR3EKNROK43O2B27S5L5U6GYB7HBVRBPQPH4WDDWZQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:14.819905028 (591h12m14.690988063s) [noise=true , address=185.167.204.206:4058 , piece_id=ROOCGK64M5RHIWNAVCSZNYTZ6M2ORWB7MJ27XJVDTFVQIKJZHY4A, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:14.820479312 (591h12m14.691562357s) [noise=true , address=185.167.204.205:4021 , piece_id=KP7J5XSY4SKSKZG7XXFZMNKMRP4472PXI7CW2UQBZICN4MAFO5RQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:14.823149296 (591h12m14.694232331s) [noise=true , address=185.167.204.205:4034 , piece_id=OWZHQB5ZDT2QZ2QAHZJVRWQ222MX6CWLKJUOIAAGSMAVLWGJAOBA, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:14.827906996 (591h12m14.698990041s) [noise=true , address=185.167.204.206:4055 , piece_id=VYEDR6XHZBZMTVJ5U5OF67WZOGVMCMQLXXBQUI7DHBCCVQ2RRERQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:14.828003769 (591h12m14.699086814s) [noise=true , address=185.167.204.205:4016 , piece_id=I6MUDLAXUUDJL2C4HFE22C2L56A7D6HSCNWXKWJ7OKO2N4RA4W7Q, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:14.828893203 (591h12m14.699976238s) [noise=true , address=185.167.204.206:4064 , piece_id=H4PP7HPHXRX2LOZ4776AMWAJDN7OC435E2K4BXI7XOZYKTB4YUNA, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:14.829477074 (591h12m14.700560109s) [noise=true , address=185.167.204.205:4017 , piece_id=CZP65KETMGOHFDMV7KENRL4V36UHBPWRDCBBPUM5NE5L2PV4AQCQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:14.829774972 (591h12m14.700858007s) [noise=true , address=185.167.204.206:4043 , piece_id=G7VZYID4TXGNPDN63NQULXOVSX42WMFFNPBX6KQJ5L2WSW4D2YGQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:14.831734493 (591h12m14.702817528s) [noise=true , address=185.167.204.206:4049 , piece_id=BQ7JEGSGPIFJUGZFBYRPDMBNPZ24BG6TEG4VUM4JUBTE4T2RF4SA, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:15.405688561 (591h12m15.276771606s) [noise=true , address=81.171.14.70:4027 , piece_id=6WEORWX5QOMFSJUEADIIBHHIPKNFP4NJPZAJADBUTPUSG7MZJMOQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:15.417110440 (591h12m15.288193485s) [noise=true , address=81.171.14.70:4031 , piece_id=U6Z4SNK32ZSWQ7NDR7ACXGRC4BADDOPR65ILBR64QLGVNDBADFFQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:15.425600446 (591h12m15.296683481s) [noise=true , address=81.171.14.70:4037 , piece_id=6A25LEIAQEJB3DNCTXQT6K7ZSKIGOW4QUGMNKLOG3CT2XXH25PSA, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:15.499056228 (591h12m15.370139263s) [noise=true , address=81.171.14.70:4038 , piece_id=3WRHW3VGXWHOEOV5Q6CLYSF7MUVWS7QHTNHLSTCKAUERPHAH3I3Q, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:15.514998925 (591h12m15.38608197s) [noise=true , address=81.171.14.70:4041 , piece_id=NORNA6WFTGPNOHU7MPTNNB5ONTBS5DJGCN2NWIZAEPYSUSIOSGEQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:15.528579544 (591h12m15.399662589s) [noise=true , address=81.171.14.70:4030 , piece_id=KITOJIT34CHS2QZBXCW2SJPZB4RHEHCMJ6IYQ4Z5KLL24YAWBIUQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:15.528991398 (591h12m15.400074434s) [noise=true , address=81.171.14.70:4047 , piece_id=ZWV5DP4YC3MSYDWEQH4ZK3VR4WZ4DUTI4VL3WVLQKFIEKIW44LOQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:15.534844544 (591h12m15.405927579s) [noise=true , address=81.171.14.70:4035 , piece_id=XCLL5SOAPJSSGF5YYQH6ZKGOHMLS6RI6H26FOW5XIJRY3TWYNEGA, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:15.915614910 (591h12m15.786697945s) [noise=true , address=81.171.14.70:4039 , piece_id=DEJDXFDHOSAMIP2FN5LJBJNBB4QLJSE2AUDW6IMA6IC4VRY7VAGQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:15.967820176 (591h12m15.838903211s) [noise=true , address=81.171.14.70:4046 , piece_id=B2MTUNKMUP5S7XW4SVQ5QI2ZO3VFB5QVAKBBY4BIZIUILM6O7VYQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:16.172116128 (591h12m16.043199163s) [noise=true , address=89.149.201.147:4007 , piece_id=HZLSYGSBP4TJH5V5XF3Q3AETJU625ERNDBD3VXW2HUZTTPAHLBMA, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:16.172137559 (591h12m16.043220604s) [seg_pos={0 0}, upload=518292]: Segment reached optimal threshold of 26 pieces.
Dec 30 06:41:16.172336949 (591h12m16.043419994s) [noise=true , address=89.149.201.147:4008 , piece_id=XYFRNJVIFFZA5S36KFPGZPVCPLRL2OW5QBJD3UR37VPPA2RX3RJA, seg_pos={0 0}, upload=518292]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QN8VP6o7qGw9Dd8EafU9xe7s7Cuf1FyqPMbnRaznrfPxfEUUn): protocol: expected piece hash; context canceled; EOF
Dec 30 06:41:16.172344213 (591h12m16.043427258s) [noise=true , address=89.149.201.147:4008 , piece_id=XYFRNJVIFFZA5S36KFPGZPVCPLRL2OW5QBJD3UR37VPPA2RX3RJA, seg_pos={0 0}, upload=518292]: Long tail cancellation due to optimal threshold
Dec 30 06:41:16.172554002 (591h12m16.043637037s) [noise=true , address=89.149.201.147:4020 , piece_id=N7G5DGEDYVKBTO5S6BYW3IYEOK7HAMK5JQIBMVPMPYRDHXODWEGA, seg_pos={0 0}, upload=518292]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12YeKHJH7CoouLFJYENJhzVLNkA32hJaqjLz3dXhr78eaSGtcic): protocol: expected piece hash; context canceled; EOF
Dec 30 06:41:16.172565233 (591h12m16.043648278s) [noise=true , address=89.149.201.147:4020 , piece_id=N7G5DGEDYVKBTO5S6BYW3IYEOK7HAMK5JQIBMVPMPYRDHXODWEGA, seg_pos={0 0}, upload=518292]: Long tail cancellation due to optimal threshold
Dec 30 06:41:16.172587506 (591h12m16.043670541s) [seg_pos={0 0}, upload=518292]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil>
Dec 30 06:41:16.172717945 (591h12m16.04380098s) [upload=518292]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 06:41:16.172724567 (591h12m16.043807602s) [upload=518292]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 06:41:16.404092237 (591h12m16.275175292s) [upload=518292]: Done uploading segments...
Dec 30 06:41:16.404119940 (591h12m16.275202975s) [upload=518292]: Upload finished. err: <nil>
Duration: 9.284901576s
core
uplink
eu-select-1
eu1-placement-15
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
15 out of 60 is failed (threshold=3) [X..X......X....X...X..X.X.X.....X.......X.X.....X....XX...X.] duration:
6.981431792s trace:
https://jaeger.dev.storj.io/trace/25efa104cb3576c8
FAIL (2025-12-30 06:48:10 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW, Piece ID: DG5N5IK35GLLLBJKWK7J53N3ER737OAGQBCHQ53AVMTIJHNWRXEA) context deadline exceeded; piecestore: (Node ID: 1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w, Piece ID: TSCFP4MGODAU4IOWHIKWBKNNAE4G2ZGCBTVRKF3KVXCCP7Z6H7EQ) context deadline exceeded
======LOGS======
30 06:47:33.902700925 (591h18m33.77378396s) [upload=518373]: Starting upload
Dec 30 06:47:33.902775336 (591h18m33.773858372s) [upload=518373]: Done starting upload
Dec 30 06:47:33.902944639 (591h18m33.774027674s) [upload=518373]: Uploading segments...
Dec 30 06:47:33.903006506 (591h18m33.774089541s) [upload=518373]: Got next segment. Inline: false
Dec 30 06:47:33.903036744 (591h18m33.774119779s) [upload=518373]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 30 06:47:33.903043197 (591h18m33.774126232s) [upload=518373]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 30 06:47:34.148006034 (591h18m34.019089069s) [seg_pos={0 0}, upload=518373]: Begin upload segment...
Dec 30 06:47:34.148062040 (591h18m34.019145075s) [seg_pos={0 0}, upload=518373]: Done begin upload segment.
Dec 30 06:47:34.148107336 (591h18m34.019190371s) [noise=true , address=45.91.116.5:4297 , piece_id=DG5N5IK35GLLLBJKWK7J53N3ER737OAGQBCHQ53AVMTIJHNWRXEA, seg_pos={0 0}, upload=518373]: Uploading piece...
Dec 30 06:47:34.148155579 (591h18m34.019238624s) [noise=true , address=145.79.227.46:4603 , piece_id=535JSH3VHKZETEICNVLFU3VCROFZVTZGOZRTN4O3UEIUDIQ6QEIQ, seg_pos={0 0}, upload=518373]: Uploading piece...
Dec 30 06:47:34.148683915 (591h18m34.01976695s) [noise=true , address=145.79.227.40:4051 , piece_id=N7JIQ4XFYDM4XQC4SMCUF2VYGGY35VA5BOKVCNAKD7BSKUSAPVNQ, seg_pos={0 0}, upload=518373]: Uploading piece...
Dec 30 06:47:34.148742016 (591h18m34.019825051s) [noise=true , address=145.79.227.48:4301 , piece_id=WMF7RPBJ3HUKXUJ63RLZFI6MVLUBJI3NLE4V7FS7NIG54XU5KLDA, seg_pos={0 0}, upload=518373]: Uploading piece...
Dec 30 06:47:34.149083746 (591h18m34.020166781s) [noise=true , address=45.91.116.4:4939 , piece_id=DRYXDGA2FTS2T5F47ZXZASU6O63V6LVCAVZPKK3OJTO5Y4IERHUQ, seg_pos={0 0}, upload=518373]: Uploading piece...
Dec 30 06:47:34.149391432 (591h18m34.020474477s) [noise=true , address=45.91.116.7:4010 , piece_id=TSCFP4MGODAU4IOWHIKWBKNNAE4G2ZGCBTVRKF3KVXCCP7Z6H7EQ, seg_pos={0 0}, upload=518373]: Uploading piece...
Dec 30 06:47:34.149687105 (591h18m34.02077014s) [noise=true , address=145.79.227.47:4265 , piece_id=FAVM24T5F2HBQIQS7IQ3GNVA5FNBCPZAUTJDVL5K4QYEGDI67MZQ, seg_pos={0 0}, upload=518373]: Uploading piece...
Dec 30 06:47:34.149987156 (591h18m34.021070191s) [noise=true , address=145.79.227.41:4085 , piece_id=V7X4XLDOB6JATX4JOWZ72XC7ONEDN64FIBBTZLJN2JLURNZRSH6Q, seg_pos={0 0}, upload=518373]: Uploading piece...
Dec 30 06:47:34.150321904 (591h18m34.021404949s) [noise=true , address=45.91.116.11:4120 , piece_id=CGPZDXJ44PHSBJ5NJUNKEUEFNYHUJPVWUETQGQ5HWAW3YZW4QDVA, seg_pos={0 0}, upload=518373]: Uploading piece...
Dec 30 06:47:34.150333786 (591h18m34.021416821s) [noise=true , address=145.79.227.39:4014 , piece_id=BLNHJN77DG2Y34Y3Z663ZUMQTBBOHWZKSW3QMBGLCVLNL6CEGJOA, seg_pos={0 0}, upload=518373]: Uploading piece...
Dec 30 06:47:34.150579524 (591h18m34.021662559s) [noise=true , address=145.79.227.19:4078 , piece_id=C2VTC5RDJBTU5Z2KKJENXLKYY2WIWBFELGDCRTJLRGSS3HWCG3XQ, seg_pos={0 0}, upload=518373]: Uploading piece...
Dec 30 06:47:34.150705945 (591h18m34.02178898s) [noise=true , address=145.79.227.32:4834 , piece_id=BQL2WD5B6ZKNH7B4AX7ELFSMKQVLVJCELGIDJYPKVSHURFJSRJLQ, seg_pos={0 0}, upload=518373]: Uploading piece...
Dec 30 06:47:34.151198954 (591h18m34.022281989s) [noise=true , address=45.91.116.14:4191 , piece_id=S6RREAK6WN5LM7T7TKBMPTDB7G2UWNGZ2L244ULRIMBEA6MOWGLA, seg_pos={0 0}, upload=518373]: Uploading piece...
Dec 30 06:47:34.151603264 (591h18m34.022686299s) [noise=true , address=145.79.227.43:4159 , piece_id=G55LMZZF32WFDATSY2BQKII65LYTA3IXEW5FIKYV444SNWQ7R2SQ, seg_pos={0 0}, upload=518373]: Uploading piece...
Dec 30 06:47:34.151984440 (591h18m34.023067485s) [noise=true , address=145.79.227.2:4114 , piece_id=QUHSCSX6TMOUGUKSUMAOENS4UBW7W4TBCTY4GAOCDH27IK2EQRBQ, seg_pos={0 0}, upload=518373]: Uploading piece...
Dec 30 06:47:34.152395842 (591h18m34.023478877s) [noise=true , address=145.79.227.17:4970 , piece_id=OR5LHPTUBOS3DDI4W2XFT5ED3Y5FGDOOVP6VPOHYYLELNS7SZB3A, seg_pos={0 0}, upload=518373]: Uploading piece...
Dec 30 06:47:35.844170154 (591h18m35.715253189s) [upload=518373]: Next returned nil segment
Dec 30 06:47:35.844218737 (591h18m35.715301792s) [upload=518373]: Waiting for error group managing segments...
Dec 30 06:47:36.127146455 (591h18m35.9982295s) [noise=true , address=145.79.227.19:4078 , piece_id=C2VTC5RDJBTU5Z2KKJENXLKYY2WIWBFELGDCRTJLRGSS3HWCG3XQ, seg_pos={0 0}, upload=518373]: Done uploading piece. err: <nil>
Dec 30 06:47:36.130047630 (591h18m36.001130665s) [noise=true , address=145.79.227.39:4014 , piece_id=BLNHJN77DG2Y34Y3Z663ZUMQTBBOHWZKSW3QMBGLCVLNL6CEGJOA, seg_pos={0 0}, upload=518373]: Done uploading piece. err: <nil>
Dec 30 06:47:36.140822898 (591h18m36.011905933s) [noise=true , address=145.79.227.2:4114 , piece_id=QUHSCSX6TMOUGUKSUMAOENS4UBW7W4TBCTY4GAOCDH27IK2EQRBQ, seg_pos={0 0}, upload=518373]: Done uploading piece. err: <nil>
Dec 30 06:47:36.141865904 (591h18m36.012948949s) [noise=true , address=145.79.227.46:4603 , piece_id=535JSH3VHKZETEICNVLFU3VCROFZVTZGOZRTN4O3UEIUDIQ6QEIQ, seg_pos={0 0}, upload=518373]: Done uploading piece. err: <nil>
Dec 30 06:47:36.142992949 (591h18m36.014075984s) [noise=true , address=145.79.227.48:4301 , piece_id=WMF7RPBJ3HUKXUJ63RLZFI6MVLUBJI3NLE4V7FS7NIG54XU5KLDA, seg_pos={0 0}, upload=518373]: Done uploading piece. err: <nil>
Dec 30 06:47:36.154997809 (591h18m36.026080844s) [noise=true , address=145.79.227.43:4159 , piece_id=G55LMZZF32WFDATSY2BQKII65LYTA3IXEW5FIKYV444SNWQ7R2SQ, seg_pos={0 0}, upload=518373]: Done uploading piece. err: <nil>
Dec 30 06:47:36.167251051 (591h18m36.038334096s) [noise=true , address=145.79.227.41:4085 , piece_id=V7X4XLDOB6JATX4JOWZ72XC7ONEDN64FIBBTZLJN2JLURNZRSH6Q, seg_pos={0 0}, upload=518373]: Done uploading piece. err: <nil>
Dec 30 06:47:36.171358592 (591h18m36.042441637s) [noise=true , address=145.79.227.47:4265 , piece_id=FAVM24T5F2HBQIQS7IQ3GNVA5FNBCPZAUTJDVL5K4QYEGDI67MZQ, seg_pos={0 0}, upload=518373]: Done uploading piece. err: <nil>
Dec 30 06:47:36.204427992 (591h18m36.075511027s) [noise=true , address=145.79.227.32:4834 , piece_id=BQL2WD5B6ZKNH7B4AX7ELFSMKQVLVJCELGIDJYPKVSHURFJSRJLQ, seg_pos={0 0}, upload=518373]: Done uploading piece. err: <nil>
Dec 30 06:47:36.219970897 (591h18m36.091053932s) [noise=true , address=145.79.227.17:4970 , piece_id=OR5LHPTUBOS3DDI4W2XFT5ED3Y5FGDOOVP6VPOHYYLELNS7SZB3A, seg_pos={0 0}, upload=518373]: Done uploading piece. err: <nil>
Dec 30 06:47:36.460845771 (591h18m36.331928816s) [noise=true , address=45.91.116.4:4939 , piece_id=DRYXDGA2FTS2T5F47ZXZASU6O63V6LVCAVZPKK3OJTO5Y4IERHUQ, seg_pos={0 0}, upload=518373]: Done uploading piece. err: <nil>
Dec 30 06:47:36.559892892 (591h18m36.430975927s) [noise=true , address=45.91.116.5:4297 , piece_id=DG5N5IK35GLLLBJKWK7J53N3ER737OAGQBCHQ53AVMTIJHNWRXEA, seg_pos={0 0}, upload=518373]: Done uploading piece. err: <nil>
Dec 30 06:47:36.564095394 (591h18m36.435178429s) [noise=true , address=45.91.116.7:4010 , piece_id=TSCFP4MGODAU4IOWHIKWBKNNAE4G2ZGCBTVRKF3KVXCCP7Z6H7EQ, seg_pos={0 0}, upload=518373]: Done uploading piece. err: <nil>
Dec 30 06:47:36.564160248 (591h18m36.435243283s) [seg_pos={0 0}, upload=518373]: Segment reached optimal threshold of 13 pieces.
Dec 30 06:47:36.564563686 (591h18m36.435646731s) [noise=true , address=45.91.116.11:4120 , piece_id=CGPZDXJ44PHSBJ5NJUNKEUEFNYHUJPVWUETQGQ5HWAW3YZW4QDVA, seg_pos={0 0}, upload=518373]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8): protocol: write tcp 188.166.225.170:50072->45.91.116.11:4120: use of closed network connection; write tcp 188.166.225.170:50072->45.91.116.11:4120: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50072->45.91.116.11:4120: use of closed network connection
Dec 30 06:47:36.564574417 (591h18m36.435657452s) [noise=true , address=45.91.116.11:4120 , piece_id=CGPZDXJ44PHSBJ5NJUNKEUEFNYHUJPVWUETQGQ5HWAW3YZW4QDVA, seg_pos={0 0}, upload=518373]: Long tail cancellation due to optimal threshold
Dec 30 06:47:36.564877924 (591h18m36.435960959s) [noise=true , address=45.91.116.14:4191 , piece_id=S6RREAK6WN5LM7T7TKBMPTDB7G2UWNGZ2L244ULRIMBEA6MOWGLA, seg_pos={0 0}, upload=518373]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq): protocol: expected piece hash; context canceled; EOF
Dec 30 06:47:36.564886651 (591h18m36.435969696s) [noise=true , address=45.91.116.14:4191 , piece_id=S6RREAK6WN5LM7T7TKBMPTDB7G2UWNGZ2L244ULRIMBEA6MOWGLA, seg_pos={0 0}, upload=518373]: Long tail cancellation due to optimal threshold
Dec 30 06:47:36.565006689 (591h18m36.436089734s) [noise=true , address=145.79.227.40:4051 , piece_id=N7JIQ4XFYDM4XQC4SMCUF2VYGGY35VA5BOKVCNAKD7BSKUSAPVNQ, seg_pos={0 0}, upload=518373]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rPbTARUrq7YsD8X75ocnyEvC517itk79s3Dp1wUg72fVuBS8n): protocol: expected piece hash; context canceled; EOF
Dec 30 06:47:36.565014714 (591h18m36.436097749s) [noise=true , address=145.79.227.40:4051 , piece_id=N7JIQ4XFYDM4XQC4SMCUF2VYGGY35VA5BOKVCNAKD7BSKUSAPVNQ, seg_pos={0 0}, upload=518373]: Long tail cancellation due to optimal threshold
Dec 30 06:47:36.565048769 (591h18m36.436131804s) [seg_pos={0 0}, upload=518373]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 30 06:47:36.565134443 (591h18m36.436217478s) [upload=518373]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 06:47:36.565140214 (591h18m36.436223259s) [upload=518373]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 06:47:36.856746186 (591h18m36.727829231s) [upload=518373]: Done uploading segments...
Dec 30 06:47:36.856766275 (591h18m36.72784932s) [upload=518373]: Upload finished. err: <nil>
Duration: 6.981447111s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
15 out of 60 is failed (threshold=3) [.X..X......X....X...X..X.X.X.....X.......X.X.....X....XX...X] duration:
30.003103442s trace:
https://jaeger.dev.storj.io/trace/25efa104cb3576c8
FAIL (2025-12-30 06:48:03 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW, Piece ID: DG5N5IK35GLLLBJKWK7J53N3ER737OAGQBCHQ53AVMTIJHNWRXEA) context deadline exceeded; piecestore: (Node ID: 1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w, Piece ID: TSCFP4MGODAU4IOWHIKWBKNNAE4G2ZGCBTVRKF3KVXCCP7Z6H7EQ) context deadline exceeded
======LOGS======
30 06:47:33.902700925 (591h18m33.77378396s) [upload=518373]: Starting upload
Dec 30 06:47:33.902775336 (591h18m33.773858372s) [upload=518373]: Done starting upload
Dec 30 06:47:33.902944639 (591h18m33.774027674s) [upload=518373]: Uploading segments...
Dec 30 06:47:33.903006506 (591h18m33.774089541s) [upload=518373]: Got next segment. Inline: false
Dec 30 06:47:33.903036744 (591h18m33.774119779s) [upload=518373]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 30 06:47:33.903043197 (591h18m33.774126232s) [upload=518373]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 30 06:47:34.148006034 (591h18m34.019089069s) [seg_pos={0 0}, upload=518373]: Begin upload segment...
Dec 30 06:47:34.148062040 (591h18m34.019145075s) [seg_pos={0 0}, upload=518373]: Done begin upload segment.
Dec 30 06:47:34.148107336 (591h18m34.019190371s) [noise=true , address=45.91.116.5:4297 , piece_id=DG5N5IK35GLLLBJKWK7J53N3ER737OAGQBCHQ53AVMTIJHNWRXEA, seg_pos={0 0}, upload=518373]: Uploading piece...
Dec 30 06:47:34.148155579 (591h18m34.019238624s) [noise=true , address=145.79.227.46:4603 , piece_id=535JSH3VHKZETEICNVLFU3VCROFZVTZGOZRTN4O3UEIUDIQ6QEIQ, seg_pos={0 0}, upload=518373]: Uploading piece...
Dec 30 06:47:34.148683915 (591h18m34.01976695s) [noise=true , address=145.79.227.40:4051 , piece_id=N7JIQ4XFYDM4XQC4SMCUF2VYGGY35VA5BOKVCNAKD7BSKUSAPVNQ, seg_pos={0 0}, upload=518373]: Uploading piece...
Dec 30 06:47:34.148742016 (591h18m34.019825051s) [noise=true , address=145.79.227.48:4301 , piece_id=WMF7RPBJ3HUKXUJ63RLZFI6MVLUBJI3NLE4V7FS7NIG54XU5KLDA, seg_pos={0 0}, upload=518373]: Uploading piece...
Dec 30 06:47:34.149083746 (591h18m34.020166781s) [noise=true , address=45.91.116.4:4939 , piece_id=DRYXDGA2FTS2T5F47ZXZASU6O63V6LVCAVZPKK3OJTO5Y4IERHUQ, seg_pos={0 0}, upload=518373]: Uploading piece...
Dec 30 06:47:34.149391432 (591h18m34.020474477s) [noise=true , address=45.91.116.7:4010 , piece_id=TSCFP4MGODAU4IOWHIKWBKNNAE4G2ZGCBTVRKF3KVXCCP7Z6H7EQ, seg_pos={0 0}, upload=518373]: Uploading piece...
Dec 30 06:47:34.149687105 (591h18m34.02077014s) [noise=true , address=145.79.227.47:4265 , piece_id=FAVM24T5F2HBQIQS7IQ3GNVA5FNBCPZAUTJDVL5K4QYEGDI67MZQ, seg_pos={0 0}, upload=518373]: Uploading piece...
Dec 30 06:47:34.149987156 (591h18m34.021070191s) [noise=true , address=145.79.227.41:4085 , piece_id=V7X4XLDOB6JATX4JOWZ72XC7ONEDN64FIBBTZLJN2JLURNZRSH6Q, seg_pos={0 0}, upload=518373]: Uploading piece...
Dec 30 06:47:34.150321904 (591h18m34.021404949s) [noise=true , address=45.91.116.11:4120 , piece_id=CGPZDXJ44PHSBJ5NJUNKEUEFNYHUJPVWUETQGQ5HWAW3YZW4QDVA, seg_pos={0 0}, upload=518373]: Uploading piece...
Dec 30 06:47:34.150333786 (591h18m34.021416821s) [noise=true , address=145.79.227.39:4014 , piece_id=BLNHJN77DG2Y34Y3Z663ZUMQTBBOHWZKSW3QMBGLCVLNL6CEGJOA, seg_pos={0 0}, upload=518373]: Uploading piece...
Dec 30 06:47:34.150579524 (591h18m34.021662559s) [noise=true , address=145.79.227.19:4078 , piece_id=C2VTC5RDJBTU5Z2KKJENXLKYY2WIWBFELGDCRTJLRGSS3HWCG3XQ, seg_pos={0 0}, upload=518373]: Uploading piece...
Dec 30 06:47:34.150705945 (591h18m34.02178898s) [noise=true , address=145.79.227.32:4834 , piece_id=BQL2WD5B6ZKNH7B4AX7ELFSMKQVLVJCELGIDJYPKVSHURFJSRJLQ, seg_pos={0 0}, upload=518373]: Uploading piece...
Dec 30 06:47:34.151198954 (591h18m34.022281989s) [noise=true , address=45.91.116.14:4191 , piece_id=S6RREAK6WN5LM7T7TKBMPTDB7G2UWNGZ2L244ULRIMBEA6MOWGLA, seg_pos={0 0}, upload=518373]: Uploading piece...
Dec 30 06:47:34.151603264 (591h18m34.022686299s) [noise=true , address=145.79.227.43:4159 , piece_id=G55LMZZF32WFDATSY2BQKII65LYTA3IXEW5FIKYV444SNWQ7R2SQ, seg_pos={0 0}, upload=518373]: Uploading piece...
Dec 30 06:47:34.151984440 (591h18m34.023067485s) [noise=true , address=145.79.227.2:4114 , piece_id=QUHSCSX6TMOUGUKSUMAOENS4UBW7W4TBCTY4GAOCDH27IK2EQRBQ, seg_pos={0 0}, upload=518373]: Uploading piece...
Dec 30 06:47:34.152395842 (591h18m34.023478877s) [noise=true , address=145.79.227.17:4970 , piece_id=OR5LHPTUBOS3DDI4W2XFT5ED3Y5FGDOOVP6VPOHYYLELNS7SZB3A, seg_pos={0 0}, upload=518373]: Uploading piece...
Dec 30 06:47:35.844170154 (591h18m35.715253189s) [upload=518373]: Next returned nil segment
Dec 30 06:47:35.844218737 (591h18m35.715301792s) [upload=518373]: Waiting for error group managing segments...
Dec 30 06:47:36.127146455 (591h18m35.9982295s) [noise=true , address=145.79.227.19:4078 , piece_id=C2VTC5RDJBTU5Z2KKJENXLKYY2WIWBFELGDCRTJLRGSS3HWCG3XQ, seg_pos={0 0}, upload=518373]: Done uploading piece. err: <nil>
Dec 30 06:47:36.130047630 (591h18m36.001130665s) [noise=true , address=145.79.227.39:4014 , piece_id=BLNHJN77DG2Y34Y3Z663ZUMQTBBOHWZKSW3QMBGLCVLNL6CEGJOA, seg_pos={0 0}, upload=518373]: Done uploading piece. err: <nil>
Dec 30 06:47:36.140822898 (591h18m36.011905933s) [noise=true , address=145.79.227.2:4114 , piece_id=QUHSCSX6TMOUGUKSUMAOENS4UBW7W4TBCTY4GAOCDH27IK2EQRBQ, seg_pos={0 0}, upload=518373]: Done uploading piece. err: <nil>
Dec 30 06:47:36.141865904 (591h18m36.012948949s) [noise=true , address=145.79.227.46:4603 , piece_id=535JSH3VHKZETEICNVLFU3VCROFZVTZGOZRTN4O3UEIUDIQ6QEIQ, seg_pos={0 0}, upload=518373]: Done uploading piece. err: <nil>
Dec 30 06:47:36.142992949 (591h18m36.014075984s) [noise=true , address=145.79.227.48:4301 , piece_id=WMF7RPBJ3HUKXUJ63RLZFI6MVLUBJI3NLE4V7FS7NIG54XU5KLDA, seg_pos={0 0}, upload=518373]: Done uploading piece. err: <nil>
Dec 30 06:47:36.154997809 (591h18m36.026080844s) [noise=true , address=145.79.227.43:4159 , piece_id=G55LMZZF32WFDATSY2BQKII65LYTA3IXEW5FIKYV444SNWQ7R2SQ, seg_pos={0 0}, upload=518373]: Done uploading piece. err: <nil>
Dec 30 06:47:36.167251051 (591h18m36.038334096s) [noise=true , address=145.79.227.41:4085 , piece_id=V7X4XLDOB6JATX4JOWZ72XC7ONEDN64FIBBTZLJN2JLURNZRSH6Q, seg_pos={0 0}, upload=518373]: Done uploading piece. err: <nil>
Dec 30 06:47:36.171358592 (591h18m36.042441637s) [noise=true , address=145.79.227.47:4265 , piece_id=FAVM24T5F2HBQIQS7IQ3GNVA5FNBCPZAUTJDVL5K4QYEGDI67MZQ, seg_pos={0 0}, upload=518373]: Done uploading piece. err: <nil>
Dec 30 06:47:36.204427992 (591h18m36.075511027s) [noise=true , address=145.79.227.32:4834 , piece_id=BQL2WD5B6ZKNH7B4AX7ELFSMKQVLVJCELGIDJYPKVSHURFJSRJLQ, seg_pos={0 0}, upload=518373]: Done uploading piece. err: <nil>
Dec 30 06:47:36.219970897 (591h18m36.091053932s) [noise=true , address=145.79.227.17:4970 , piece_id=OR5LHPTUBOS3DDI4W2XFT5ED3Y5FGDOOVP6VPOHYYLELNS7SZB3A, seg_pos={0 0}, upload=518373]: Done uploading piece. err: <nil>
Dec 30 06:47:36.460845771 (591h18m36.331928816s) [noise=true , address=45.91.116.4:4939 , piece_id=DRYXDGA2FTS2T5F47ZXZASU6O63V6LVCAVZPKK3OJTO5Y4IERHUQ, seg_pos={0 0}, upload=518373]: Done uploading piece. err: <nil>
Dec 30 06:47:36.559892892 (591h18m36.430975927s) [noise=true , address=45.91.116.5:4297 , piece_id=DG5N5IK35GLLLBJKWK7J53N3ER737OAGQBCHQ53AVMTIJHNWRXEA, seg_pos={0 0}, upload=518373]: Done uploading piece. err: <nil>
Dec 30 06:47:36.564095394 (591h18m36.435178429s) [noise=true , address=45.91.116.7:4010 , piece_id=TSCFP4MGODAU4IOWHIKWBKNNAE4G2ZGCBTVRKF3KVXCCP7Z6H7EQ, seg_pos={0 0}, upload=518373]: Done uploading piece. err: <nil>
Dec 30 06:47:36.564160248 (591h18m36.435243283s) [seg_pos={0 0}, upload=518373]: Segment reached optimal threshold of 13 pieces.
Dec 30 06:47:36.564563686 (591h18m36.435646731s) [noise=true , address=45.91.116.11:4120 , piece_id=CGPZDXJ44PHSBJ5NJUNKEUEFNYHUJPVWUETQGQ5HWAW3YZW4QDVA, seg_pos={0 0}, upload=518373]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8): protocol: write tcp 188.166.225.170:50072->45.91.116.11:4120: use of closed network connection; write tcp 188.166.225.170:50072->45.91.116.11:4120: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50072->45.91.116.11:4120: use of closed network connection
Dec 30 06:47:36.564574417 (591h18m36.435657452s) [noise=true , address=45.91.116.11:4120 , piece_id=CGPZDXJ44PHSBJ5NJUNKEUEFNYHUJPVWUETQGQ5HWAW3YZW4QDVA, seg_pos={0 0}, upload=518373]: Long tail cancellation due to optimal threshold
Dec 30 06:47:36.564877924 (591h18m36.435960959s) [noise=true , address=45.91.116.14:4191 , piece_id=S6RREAK6WN5LM7T7TKBMPTDB7G2UWNGZ2L244ULRIMBEA6MOWGLA, seg_pos={0 0}, upload=518373]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq): protocol: expected piece hash; context canceled; EOF
Dec 30 06:47:36.564886651 (591h18m36.435969696s) [noise=true , address=45.91.116.14:4191 , piece_id=S6RREAK6WN5LM7T7TKBMPTDB7G2UWNGZ2L244ULRIMBEA6MOWGLA, seg_pos={0 0}, upload=518373]: Long tail cancellation due to optimal threshold
Dec 30 06:47:36.565006689 (591h18m36.436089734s) [noise=true , address=145.79.227.40:4051 , piece_id=N7JIQ4XFYDM4XQC4SMCUF2VYGGY35VA5BOKVCNAKD7BSKUSAPVNQ, seg_pos={0 0}, upload=518373]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rPbTARUrq7YsD8X75ocnyEvC517itk79s3Dp1wUg72fVuBS8n): protocol: expected piece hash; context canceled; EOF
Dec 30 06:47:36.565014714 (591h18m36.436097749s) [noise=true , address=145.79.227.40:4051 , piece_id=N7JIQ4XFYDM4XQC4SMCUF2VYGGY35VA5BOKVCNAKD7BSKUSAPVNQ, seg_pos={0 0}, upload=518373]: Long tail cancellation due to optimal threshold
Dec 30 06:47:36.565048769 (591h18m36.436131804s) [seg_pos={0 0}, upload=518373]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 30 06:47:36.565134443 (591h18m36.436217478s) [upload=518373]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 06:47:36.565140214 (591h18m36.436223259s) [upload=518373]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 06:47:36.856746186 (591h18m36.727829231s) [upload=518373]: Done uploading segments...
Dec 30 06:47:36.856766275 (591h18m36.72784932s) [upload=518373]: Upload finished. err: <nil>
Duration: 30.003162414s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)antiblip:
1 out of 60 is failed (threshold=3) [...................................................X........] duration:
6.038034736s trace:
https://jaeger.dev.storj.io/trace/4c2f4efaff0e0ea0
FAIL (2025-12-30 06:47:51 UTC)
orbiter warning:
operation: DeleteObject
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
load=518292]: Uploading piece...
Dec 30 06:41:13.598100152 (591h12m13.469183187s) [noise=true , address=81.171.14.70:4041 , piece_id=NORNA6WFTGPNOHU7MPTNNB5ONTBS5DJGCN2NWIZAEPYSUSIOSGEQ, seg_pos={0 0}, upload=518292]: Uploading piece...
Dec 30 06:41:13.598587971 (591h12m13.469671016s) [noise=true , address=89.149.201.147:4007 , piece_id=HZLSYGSBP4TJH5V5XF3Q3AETJU625ERNDBD3VXW2HUZTTPAHLBMA, seg_pos={0 0}, upload=518292]: Uploading piece...
Dec 30 06:41:13.599093143 (591h12m13.470176178s) [noise=true , address=89.149.201.147:4008 , piece_id=XYFRNJVIFFZA5S36KFPGZPVCPLRL2OW5QBJD3UR37VPPA2RX3RJA, seg_pos={0 0}, upload=518292]: Uploading piece...
Dec 30 06:41:13.599699999 (591h12m13.470783034s) [noise=true , address=81.171.14.70:4039 , piece_id=DEJDXFDHOSAMIP2FN5LJBJNBB4QLJSE2AUDW6IMA6IC4VRY7VAGQ, seg_pos={0 0}, upload=518292]: Uploading piece...
Dec 30 06:41:13.600136169 (591h12m13.471219204s) [noise=true , address=185.167.204.205:4034 , piece_id=OWZHQB5ZDT2QZ2QAHZJVRWQ222MX6CWLKJUOIAAGSMAVLWGJAOBA, seg_pos={0 0}, upload=518292]: Uploading piece...
Dec 30 06:41:13.600543184 (591h12m13.471626229s) [noise=true , address=185.167.204.206:4058 , piece_id=ROOCGK64M5RHIWNAVCSZNYTZ6M2ORWB7MJ27XJVDTFVQIKJZHY4A, seg_pos={0 0}, upload=518292]: Uploading piece...
Dec 30 06:41:13.601048286 (591h12m13.472131332s) [noise=true , address=81.171.14.70:4046 , piece_id=B2MTUNKMUP5S7XW4SVQ5QI2ZO3VFB5QVAKBBY4BIZIUILM6O7VYQ, seg_pos={0 0}, upload=518292]: Uploading piece...
Dec 30 06:41:13.601698784 (591h12m13.472781819s) [noise=true , address=185.167.204.206:4064 , piece_id=H4PP7HPHXRX2LOZ4776AMWAJDN7OC435E2K4BXI7XOZYKTB4YUNA, seg_pos={0 0}, upload=518292]: Uploading piece...
Dec 30 06:41:13.602168259 (591h12m13.473251304s) [noise=true , address=185.167.204.206:4065 , piece_id=LD3J4656Q5QEUCMJ5OKEYATEVRIKSXO7XGARTN25BVM7FDIYOWKA, seg_pos={0 0}, upload=518292]: Uploading piece...
Dec 30 06:41:13.602782708 (591h12m13.473865753s) [noise=true , address=81.171.14.70:4027 , piece_id=6WEORWX5QOMFSJUEADIIBHHIPKNFP4NJPZAJADBUTPUSG7MZJMOQ, seg_pos={0 0}, upload=518292]: Uploading piece...
Dec 30 06:41:14.423395002 (591h12m14.294478047s) [upload=518292]: Next returned nil segment
Dec 30 06:41:14.423414068 (591h12m14.294497103s) [upload=518292]: Waiting for error group managing segments...
Dec 30 06:41:14.817650316 (591h12m14.688733351s) [noise=true , address=185.167.204.205:4029 , piece_id=KUOMMYJ2F4NIK6VN4C5M3Y3TMHYEZH7QFNJ6K3CLOAVI5R6VKTHQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:14.817870054 (591h12m14.688953099s) [noise=true , address=185.167.204.205:4019 , piece_id=H6AH654VGX6NHBUKODIWGL7I2RKE3DDMY4UM6VDONP2EP3NPDJBA, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:14.818623698 (591h12m14.689706744s) [noise=true , address=185.167.204.206:4065 , piece_id=LD3J4656Q5QEUCMJ5OKEYATEVRIKSXO7XGARTN25BVM7FDIYOWKA, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:14.818932436 (591h12m14.690015481s) [noise=true , address=185.167.204.206:4061 , piece_id=T2XS4IFPQSVS3T45SPQ5ZDDVT2VMTMXHKL2ZMX4OIOPRR67SCD2Q, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:14.819147046 (591h12m14.690230091s) [noise=true , address=185.167.204.206:4071 , piece_id=QI2TOY5KCDZVWCCLMAJXVUWD2HB7S7IOTEAV2EHAMUNHKOFKTNYA, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:14.819678578 (591h12m14.690761623s) [noise=true , address=185.167.204.206:4059 , piece_id=LMOGTWX2YIRR3EKNROK43O2B27S5L5U6GYB7HBVRBPQPH4WDDWZQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:14.819905028 (591h12m14.690988063s) [noise=true , address=185.167.204.206:4058 , piece_id=ROOCGK64M5RHIWNAVCSZNYTZ6M2ORWB7MJ27XJVDTFVQIKJZHY4A, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:14.820479312 (591h12m14.691562357s) [noise=true , address=185.167.204.205:4021 , piece_id=KP7J5XSY4SKSKZG7XXFZMNKMRP4472PXI7CW2UQBZICN4MAFO5RQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:14.823149296 (591h12m14.694232331s) [noise=true , address=185.167.204.205:4034 , piece_id=OWZHQB5ZDT2QZ2QAHZJVRWQ222MX6CWLKJUOIAAGSMAVLWGJAOBA, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:14.827906996 (591h12m14.698990041s) [noise=true , address=185.167.204.206:4055 , piece_id=VYEDR6XHZBZMTVJ5U5OF67WZOGVMCMQLXXBQUI7DHBCCVQ2RRERQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:14.828003769 (591h12m14.699086814s) [noise=true , address=185.167.204.205:4016 , piece_id=I6MUDLAXUUDJL2C4HFE22C2L56A7D6HSCNWXKWJ7OKO2N4RA4W7Q, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:14.828893203 (591h12m14.699976238s) [noise=true , address=185.167.204.206:4064 , piece_id=H4PP7HPHXRX2LOZ4776AMWAJDN7OC435E2K4BXI7XOZYKTB4YUNA, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:14.829477074 (591h12m14.700560109s) [noise=true , address=185.167.204.205:4017 , piece_id=CZP65KETMGOHFDMV7KENRL4V36UHBPWRDCBBPUM5NE5L2PV4AQCQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:14.829774972 (591h12m14.700858007s) [noise=true , address=185.167.204.206:4043 , piece_id=G7VZYID4TXGNPDN63NQULXOVSX42WMFFNPBX6KQJ5L2WSW4D2YGQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:14.831734493 (591h12m14.702817528s) [noise=true , address=185.167.204.206:4049 , piece_id=BQ7JEGSGPIFJUGZFBYRPDMBNPZ24BG6TEG4VUM4JUBTE4T2RF4SA, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:15.405688561 (591h12m15.276771606s) [noise=true , address=81.171.14.70:4027 , piece_id=6WEORWX5QOMFSJUEADIIBHHIPKNFP4NJPZAJADBUTPUSG7MZJMOQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:15.417110440 (591h12m15.288193485s) [noise=true , address=81.171.14.70:4031 , piece_id=U6Z4SNK32ZSWQ7NDR7ACXGRC4BADDOPR65ILBR64QLGVNDBADFFQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:15.425600446 (591h12m15.296683481s) [noise=true , address=81.171.14.70:4037 , piece_id=6A25LEIAQEJB3DNCTXQT6K7ZSKIGOW4QUGMNKLOG3CT2XXH25PSA, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:15.499056228 (591h12m15.370139263s) [noise=true , address=81.171.14.70:4038 , piece_id=3WRHW3VGXWHOEOV5Q6CLYSF7MUVWS7QHTNHLSTCKAUERPHAH3I3Q, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:15.514998925 (591h12m15.38608197s) [noise=true , address=81.171.14.70:4041 , piece_id=NORNA6WFTGPNOHU7MPTNNB5ONTBS5DJGCN2NWIZAEPYSUSIOSGEQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:15.528579544 (591h12m15.399662589s) [noise=true , address=81.171.14.70:4030 , piece_id=KITOJIT34CHS2QZBXCW2SJPZB4RHEHCMJ6IYQ4Z5KLL24YAWBIUQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:15.528991398 (591h12m15.400074434s) [noise=true , address=81.171.14.70:4047 , piece_id=ZWV5DP4YC3MSYDWEQH4ZK3VR4WZ4DUTI4VL3WVLQKFIEKIW44LOQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:15.534844544 (591h12m15.405927579s) [noise=true , address=81.171.14.70:4035 , piece_id=XCLL5SOAPJSSGF5YYQH6ZKGOHMLS6RI6H26FOW5XIJRY3TWYNEGA, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:15.915614910 (591h12m15.786697945s) [noise=true , address=81.171.14.70:4039 , piece_id=DEJDXFDHOSAMIP2FN5LJBJNBB4QLJSE2AUDW6IMA6IC4VRY7VAGQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:15.967820176 (591h12m15.838903211s) [noise=true , address=81.171.14.70:4046 , piece_id=B2MTUNKMUP5S7XW4SVQ5QI2ZO3VFB5QVAKBBY4BIZIUILM6O7VYQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:16.172116128 (591h12m16.043199163s) [noise=true , address=89.149.201.147:4007 , piece_id=HZLSYGSBP4TJH5V5XF3Q3AETJU625ERNDBD3VXW2HUZTTPAHLBMA, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:16.172137559 (591h12m16.043220604s) [seg_pos={0 0}, upload=518292]: Segment reached optimal threshold of 26 pieces.
Dec 30 06:41:16.172336949 (591h12m16.043419994s) [noise=true , address=89.149.201.147:4008 , piece_id=XYFRNJVIFFZA5S36KFPGZPVCPLRL2OW5QBJD3UR37VPPA2RX3RJA, seg_pos={0 0}, upload=518292]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QN8VP6o7qGw9Dd8EafU9xe7s7Cuf1FyqPMbnRaznrfPxfEUUn): protocol: expected piece hash; context canceled; EOF
Dec 30 06:41:16.172344213 (591h12m16.043427258s) [noise=true , address=89.149.201.147:4008 , piece_id=XYFRNJVIFFZA5S36KFPGZPVCPLRL2OW5QBJD3UR37VPPA2RX3RJA, seg_pos={0 0}, upload=518292]: Long tail cancellation due to optimal threshold
Dec 30 06:41:16.172554002 (591h12m16.043637037s) [noise=true , address=89.149.201.147:4020 , piece_id=N7G5DGEDYVKBTO5S6BYW3IYEOK7HAMK5JQIBMVPMPYRDHXODWEGA, seg_pos={0 0}, upload=518292]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12YeKHJH7CoouLFJYENJhzVLNkA32hJaqjLz3dXhr78eaSGtcic): protocol: expected piece hash; context canceled; EOF
Dec 30 06:41:16.172565233 (591h12m16.043648278s) [noise=true , address=89.149.201.147:4020 , piece_id=N7G5DGEDYVKBTO5S6BYW3IYEOK7HAMK5JQIBMVPMPYRDHXODWEGA, seg_pos={0 0}, upload=518292]: Long tail cancellation due to optimal threshold
Dec 30 06:41:16.172587506 (591h12m16.043670541s) [seg_pos={0 0}, upload=518292]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil>
Dec 30 06:41:16.172717945 (591h12m16.04380098s) [upload=518292]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 06:41:16.172724567 (591h12m16.043807602s) [upload=518292]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 06:41:16.404092237 (591h12m16.275175292s) [upload=518292]: Done uploading segments...
Dec 30 06:41:16.404119940 (591h12m16.275202975s) [upload=518292]: Upload finished. err: <nil>
Duration: 6.038060666s
core
uplink
eu-select-1
eu1-placement-15
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.............X..............................................] duration:
10.947825983s trace:
https://jaeger.dev.storj.io/trace/deb96436003dee
FAIL (2025-12-30 06:47:47 UTC)
orbiter warning: read tcp 188.166.225.170:52458->45.38.24.2:443: read: connection reset by peer
Duration: 10.947869656s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)antiblip:
1 out of 60 is failed (threshold=3) [....................................................X.......] duration:
6.618149474s trace:
https://jaeger.dev.storj.io/trace/4c2f4efaff0e0ea0
FAIL (2025-12-30 06:47:31 UTC)
orbiter warning:
operation: DeleteObject
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
load=518292]: Uploading piece...
Dec 30 06:41:13.598100152 (591h12m13.469183187s) [noise=true , address=81.171.14.70:4041 , piece_id=NORNA6WFTGPNOHU7MPTNNB5ONTBS5DJGCN2NWIZAEPYSUSIOSGEQ, seg_pos={0 0}, upload=518292]: Uploading piece...
Dec 30 06:41:13.598587971 (591h12m13.469671016s) [noise=true , address=89.149.201.147:4007 , piece_id=HZLSYGSBP4TJH5V5XF3Q3AETJU625ERNDBD3VXW2HUZTTPAHLBMA, seg_pos={0 0}, upload=518292]: Uploading piece...
Dec 30 06:41:13.599093143 (591h12m13.470176178s) [noise=true , address=89.149.201.147:4008 , piece_id=XYFRNJVIFFZA5S36KFPGZPVCPLRL2OW5QBJD3UR37VPPA2RX3RJA, seg_pos={0 0}, upload=518292]: Uploading piece...
Dec 30 06:41:13.599699999 (591h12m13.470783034s) [noise=true , address=81.171.14.70:4039 , piece_id=DEJDXFDHOSAMIP2FN5LJBJNBB4QLJSE2AUDW6IMA6IC4VRY7VAGQ, seg_pos={0 0}, upload=518292]: Uploading piece...
Dec 30 06:41:13.600136169 (591h12m13.471219204s) [noise=true , address=185.167.204.205:4034 , piece_id=OWZHQB5ZDT2QZ2QAHZJVRWQ222MX6CWLKJUOIAAGSMAVLWGJAOBA, seg_pos={0 0}, upload=518292]: Uploading piece...
Dec 30 06:41:13.600543184 (591h12m13.471626229s) [noise=true , address=185.167.204.206:4058 , piece_id=ROOCGK64M5RHIWNAVCSZNYTZ6M2ORWB7MJ27XJVDTFVQIKJZHY4A, seg_pos={0 0}, upload=518292]: Uploading piece...
Dec 30 06:41:13.601048286 (591h12m13.472131332s) [noise=true , address=81.171.14.70:4046 , piece_id=B2MTUNKMUP5S7XW4SVQ5QI2ZO3VFB5QVAKBBY4BIZIUILM6O7VYQ, seg_pos={0 0}, upload=518292]: Uploading piece...
Dec 30 06:41:13.601698784 (591h12m13.472781819s) [noise=true , address=185.167.204.206:4064 , piece_id=H4PP7HPHXRX2LOZ4776AMWAJDN7OC435E2K4BXI7XOZYKTB4YUNA, seg_pos={0 0}, upload=518292]: Uploading piece...
Dec 30 06:41:13.602168259 (591h12m13.473251304s) [noise=true , address=185.167.204.206:4065 , piece_id=LD3J4656Q5QEUCMJ5OKEYATEVRIKSXO7XGARTN25BVM7FDIYOWKA, seg_pos={0 0}, upload=518292]: Uploading piece...
Dec 30 06:41:13.602782708 (591h12m13.473865753s) [noise=true , address=81.171.14.70:4027 , piece_id=6WEORWX5QOMFSJUEADIIBHHIPKNFP4NJPZAJADBUTPUSG7MZJMOQ, seg_pos={0 0}, upload=518292]: Uploading piece...
Dec 30 06:41:14.423395002 (591h12m14.294478047s) [upload=518292]: Next returned nil segment
Dec 30 06:41:14.423414068 (591h12m14.294497103s) [upload=518292]: Waiting for error group managing segments...
Dec 30 06:41:14.817650316 (591h12m14.688733351s) [noise=true , address=185.167.204.205:4029 , piece_id=KUOMMYJ2F4NIK6VN4C5M3Y3TMHYEZH7QFNJ6K3CLOAVI5R6VKTHQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:14.817870054 (591h12m14.688953099s) [noise=true , address=185.167.204.205:4019 , piece_id=H6AH654VGX6NHBUKODIWGL7I2RKE3DDMY4UM6VDONP2EP3NPDJBA, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:14.818623698 (591h12m14.689706744s) [noise=true , address=185.167.204.206:4065 , piece_id=LD3J4656Q5QEUCMJ5OKEYATEVRIKSXO7XGARTN25BVM7FDIYOWKA, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:14.818932436 (591h12m14.690015481s) [noise=true , address=185.167.204.206:4061 , piece_id=T2XS4IFPQSVS3T45SPQ5ZDDVT2VMTMXHKL2ZMX4OIOPRR67SCD2Q, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:14.819147046 (591h12m14.690230091s) [noise=true , address=185.167.204.206:4071 , piece_id=QI2TOY5KCDZVWCCLMAJXVUWD2HB7S7IOTEAV2EHAMUNHKOFKTNYA, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:14.819678578 (591h12m14.690761623s) [noise=true , address=185.167.204.206:4059 , piece_id=LMOGTWX2YIRR3EKNROK43O2B27S5L5U6GYB7HBVRBPQPH4WDDWZQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:14.819905028 (591h12m14.690988063s) [noise=true , address=185.167.204.206:4058 , piece_id=ROOCGK64M5RHIWNAVCSZNYTZ6M2ORWB7MJ27XJVDTFVQIKJZHY4A, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:14.820479312 (591h12m14.691562357s) [noise=true , address=185.167.204.205:4021 , piece_id=KP7J5XSY4SKSKZG7XXFZMNKMRP4472PXI7CW2UQBZICN4MAFO5RQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:14.823149296 (591h12m14.694232331s) [noise=true , address=185.167.204.205:4034 , piece_id=OWZHQB5ZDT2QZ2QAHZJVRWQ222MX6CWLKJUOIAAGSMAVLWGJAOBA, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:14.827906996 (591h12m14.698990041s) [noise=true , address=185.167.204.206:4055 , piece_id=VYEDR6XHZBZMTVJ5U5OF67WZOGVMCMQLXXBQUI7DHBCCVQ2RRERQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:14.828003769 (591h12m14.699086814s) [noise=true , address=185.167.204.205:4016 , piece_id=I6MUDLAXUUDJL2C4HFE22C2L56A7D6HSCNWXKWJ7OKO2N4RA4W7Q, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:14.828893203 (591h12m14.699976238s) [noise=true , address=185.167.204.206:4064 , piece_id=H4PP7HPHXRX2LOZ4776AMWAJDN7OC435E2K4BXI7XOZYKTB4YUNA, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:14.829477074 (591h12m14.700560109s) [noise=true , address=185.167.204.205:4017 , piece_id=CZP65KETMGOHFDMV7KENRL4V36UHBPWRDCBBPUM5NE5L2PV4AQCQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:14.829774972 (591h12m14.700858007s) [noise=true , address=185.167.204.206:4043 , piece_id=G7VZYID4TXGNPDN63NQULXOVSX42WMFFNPBX6KQJ5L2WSW4D2YGQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:14.831734493 (591h12m14.702817528s) [noise=true , address=185.167.204.206:4049 , piece_id=BQ7JEGSGPIFJUGZFBYRPDMBNPZ24BG6TEG4VUM4JUBTE4T2RF4SA, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:15.405688561 (591h12m15.276771606s) [noise=true , address=81.171.14.70:4027 , piece_id=6WEORWX5QOMFSJUEADIIBHHIPKNFP4NJPZAJADBUTPUSG7MZJMOQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:15.417110440 (591h12m15.288193485s) [noise=true , address=81.171.14.70:4031 , piece_id=U6Z4SNK32ZSWQ7NDR7ACXGRC4BADDOPR65ILBR64QLGVNDBADFFQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:15.425600446 (591h12m15.296683481s) [noise=true , address=81.171.14.70:4037 , piece_id=6A25LEIAQEJB3DNCTXQT6K7ZSKIGOW4QUGMNKLOG3CT2XXH25PSA, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:15.499056228 (591h12m15.370139263s) [noise=true , address=81.171.14.70:4038 , piece_id=3WRHW3VGXWHOEOV5Q6CLYSF7MUVWS7QHTNHLSTCKAUERPHAH3I3Q, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:15.514998925 (591h12m15.38608197s) [noise=true , address=81.171.14.70:4041 , piece_id=NORNA6WFTGPNOHU7MPTNNB5ONTBS5DJGCN2NWIZAEPYSUSIOSGEQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:15.528579544 (591h12m15.399662589s) [noise=true , address=81.171.14.70:4030 , piece_id=KITOJIT34CHS2QZBXCW2SJPZB4RHEHCMJ6IYQ4Z5KLL24YAWBIUQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:15.528991398 (591h12m15.400074434s) [noise=true , address=81.171.14.70:4047 , piece_id=ZWV5DP4YC3MSYDWEQH4ZK3VR4WZ4DUTI4VL3WVLQKFIEKIW44LOQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:15.534844544 (591h12m15.405927579s) [noise=true , address=81.171.14.70:4035 , piece_id=XCLL5SOAPJSSGF5YYQH6ZKGOHMLS6RI6H26FOW5XIJRY3TWYNEGA, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:15.915614910 (591h12m15.786697945s) [noise=true , address=81.171.14.70:4039 , piece_id=DEJDXFDHOSAMIP2FN5LJBJNBB4QLJSE2AUDW6IMA6IC4VRY7VAGQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:15.967820176 (591h12m15.838903211s) [noise=true , address=81.171.14.70:4046 , piece_id=B2MTUNKMUP5S7XW4SVQ5QI2ZO3VFB5QVAKBBY4BIZIUILM6O7VYQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:16.172116128 (591h12m16.043199163s) [noise=true , address=89.149.201.147:4007 , piece_id=HZLSYGSBP4TJH5V5XF3Q3AETJU625ERNDBD3VXW2HUZTTPAHLBMA, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:16.172137559 (591h12m16.043220604s) [seg_pos={0 0}, upload=518292]: Segment reached optimal threshold of 26 pieces.
Dec 30 06:41:16.172336949 (591h12m16.043419994s) [noise=true , address=89.149.201.147:4008 , piece_id=XYFRNJVIFFZA5S36KFPGZPVCPLRL2OW5QBJD3UR37VPPA2RX3RJA, seg_pos={0 0}, upload=518292]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QN8VP6o7qGw9Dd8EafU9xe7s7Cuf1FyqPMbnRaznrfPxfEUUn): protocol: expected piece hash; context canceled; EOF
Dec 30 06:41:16.172344213 (591h12m16.043427258s) [noise=true , address=89.149.201.147:4008 , piece_id=XYFRNJVIFFZA5S36KFPGZPVCPLRL2OW5QBJD3UR37VPPA2RX3RJA, seg_pos={0 0}, upload=518292]: Long tail cancellation due to optimal threshold
Dec 30 06:41:16.172554002 (591h12m16.043637037s) [noise=true , address=89.149.201.147:4020 , piece_id=N7G5DGEDYVKBTO5S6BYW3IYEOK7HAMK5JQIBMVPMPYRDHXODWEGA, seg_pos={0 0}, upload=518292]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12YeKHJH7CoouLFJYENJhzVLNkA32hJaqjLz3dXhr78eaSGtcic): protocol: expected piece hash; context canceled; EOF
Dec 30 06:41:16.172565233 (591h12m16.043648278s) [noise=true , address=89.149.201.147:4020 , piece_id=N7G5DGEDYVKBTO5S6BYW3IYEOK7HAMK5JQIBMVPMPYRDHXODWEGA, seg_pos={0 0}, upload=518292]: Long tail cancellation due to optimal threshold
Dec 30 06:41:16.172587506 (591h12m16.043670541s) [seg_pos={0 0}, upload=518292]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil>
Dec 30 06:41:16.172717945 (591h12m16.04380098s) [upload=518292]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 06:41:16.172724567 (591h12m16.043807602s) [upload=518292]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 06:41:16.404092237 (591h12m16.275175292s) [upload=518292]: Done uploading segments...
Dec 30 06:41:16.404119940 (591h12m16.275202975s) [upload=518292]: Upload finished. err: <nil>
Duration: 6.618179672s
core
uplink
eu-select-1
eu1-placement-15
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
10 out of 60 is failed (threshold=6) [..X...........X..............XX...X......X..X..X.....X.....X] duration:
5.001538144s trace:
https://jaeger.dev.storj.io/trace/2dfc5fc3a860afad
FAIL (2025-12-30 06:47:25 UTC)
orbiter warning: context deadline exceeded
Duration: 5.00156191s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
4 out of 60 is failed (threshold=3) [....X...X......X............X...............................] duration:
40.778875984s trace:
https://jaeger.dev.storj.io/trace/45b769fc71e13448
FAIL (2025-12-30 06:47:18 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: context deadline exceeded
======LOGS======
.577697683 (590h53m11.448780728s) [noise=true , address=193.149.164.143:4704 , piece_id=4T2IJIA6WHI5LMIUP5BLJLCD3GRY3A6PXR6G5OWJWUSYZT5ELGHQ, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.578173528 (590h53m11.449256573s) [noise=true , address=64.49.13.27:4965 , piece_id=CO4O5F2IBZAAKBZBV3HWNNOXZOGLNNDPALWMQZ3DEM3MREASVFEA, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.578542992 (590h53m11.449626027s) [noise=true , address=77.90.31.15:4729 , piece_id=NVMXFEILHGT622IIPYYT4FFHC2WZLHMCZM4VT5WUPHRAPH3DX6TA, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.578846911 (590h53m11.449929946s) [noise=true , address=145.79.227.43:4177 , piece_id=YM37ZI3WNZ7SXGTFC224TOVVTHYNFIPRQB2TFLEZ735RNZE553FQ, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.579458044 (590h53m11.450541089s) [noise=true , address=45.91.116.11:4150 , piece_id=E66IDHTJDLYLYJIOWMOIKOOWLXCUNCMBIQ7SQBAD3VTJF6VX5JLQ, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.579891209 (590h53m11.450974245s) [noise=true , address=145.79.227.29:4945 , piece_id=SHL73IDEI54GRCWYNWHRUSJZZ4UUXREAX54DFJLTG5QPZCF5QKIQ, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.579915535 (590h53m11.45099858s) [noise=true , address=145.79.227.30:4772 , piece_id=PVMD5DWRRPVARTSZ6O7HN6MUVTZUYWLYSNQR5GDMNKUJZGXMYPFQ, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.580325466 (590h53m11.451408511s) [noise=true , address=77.90.31.12:4603 , piece_id=2I4L7F2WEGK5OSLODJSJ4672URBA7DUGXWZKTYB6ZNA2TQIGVXWA, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.580379599 (590h53m11.451462644s) [noise=true , address=77.90.31.14:4671 , piece_id=263TRISJ5ADFTX43ULSSGBAF2KMOXAPISJ56LLE53D74XOEMQ5EA, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.580692044 (590h53m11.451775089s) [noise=true , address=145.79.227.3:4175 , piece_id=UCV5WQNVWEAMEDBWDNXEVSQEGXPJE7KD6YSWC5A4WXYUKUCK4NLA, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.580857859 (590h53m11.451940894s) [noise=true , address=77.90.31.19:4860 , piece_id=U2MNFOAD7FHCDBE4FUYOROIP3P33ROJ7K75SF5Z747EFKFTF7RGA, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.581089210 (590h53m11.452172255s) [noise=true , address=145.79.227.44:4203 , piece_id=WHCKQ2FRLYAZE7Q6HYZNTS4EISVHZNAPDDKH34WN33ALK4MVK2EA, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.581351320 (590h53m11.452434355s) [noise=true , address=64.49.13.154:4928 , piece_id=SEVBZBMNGURP5TRUEBA4PHRSA6MNANVJUKINALJUWPQWBGKVYNTA, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.581457091 (590h53m11.452540126s) [noise=true , address=145.79.227.34:4909 , piece_id=3QSWTNR5LGUUAY3N5HJKSPL3TRQ6WO5F5OD4VTXKWHWUXUF7PVVQ, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.581863144 (590h53m11.452946179s) [noise=true , address=145.79.227.33:4881 , piece_id=EX66JF7U7A4ZSRWJMP4MCLQRPW2ZZFZFIUNV72XZXU7E5O3KWX3A, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.584010443 (590h53m11.455093478s) [noise=true , address=145.79.227.40:4070 , piece_id=YOAGDXNN4TTCK35LLQDSP24BOWLZXGNSQU7WMKEEVVMPXDVI74JA, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.585122791 (590h53m11.456205826s) [noise=true , address=45.91.116.14:4217 , piece_id=L4MAHTGKTNWBSN3YTB74E3DBJWVMA3DPU57MQK5FNK35DXRZ6AIA, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:12.983305402 (590h53m12.854388447s) [upload=517998]: Next returned nil segment
Dec 30 06:22:12.983321313 (590h53m12.854404348s) [upload=517998]: Waiting for error group managing segments...
Dec 30 06:22:13.379703053 (590h53m13.250786088s) [noise=true , address=64.49.13.27:4965 , piece_id=CO4O5F2IBZAAKBZBV3HWNNOXZOGLNNDPALWMQZ3DEM3MREASVFEA, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:13.418055622 (590h53m13.289138657s) [noise=true , address=145.79.227.3:4175 , piece_id=UCV5WQNVWEAMEDBWDNXEVSQEGXPJE7KD6YSWC5A4WXYUKUCK4NLA, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:13.509034160 (590h53m13.380117205s) [noise=true , address=64.49.13.154:4928 , piece_id=SEVBZBMNGURP5TRUEBA4PHRSA6MNANVJUKINALJUWPQWBGKVYNTA, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:13.642886154 (590h53m13.513969199s) [noise=true , address=77.90.31.14:4671 , piece_id=263TRISJ5ADFTX43ULSSGBAF2KMOXAPISJ56LLE53D74XOEMQ5EA, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:13.788145520 (590h53m13.659228555s) [noise=true , address=77.90.31.12:4603 , piece_id=2I4L7F2WEGK5OSLODJSJ4672URBA7DUGXWZKTYB6ZNA2TQIGVXWA, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:13.858878757 (590h53m13.729961792s) [noise=true , address=145.79.227.49:4369 , piece_id=XUEXGNJJIJ4DUBDNUV7CKF6J5JYHSLZAEPVGEDGBUBY42BIP4XEQ, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:13.879550569 (590h53m13.750633614s) [noise=true , address=145.79.227.34:4909 , piece_id=3QSWTNR5LGUUAY3N5HJKSPL3TRQ6WO5F5OD4VTXKWHWUXUF7PVVQ, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:13.940897723 (590h53m13.811980769s) [noise=true , address=77.90.31.19:4860 , piece_id=U2MNFOAD7FHCDBE4FUYOROIP3P33ROJ7K75SF5Z747EFKFTF7RGA, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:13.989791936 (590h53m13.860874981s) [noise=true , address=45.91.116.11:4150 , piece_id=E66IDHTJDLYLYJIOWMOIKOOWLXCUNCMBIQ7SQBAD3VTJF6VX5JLQ, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:14.289516718 (590h53m14.160599753s) [noise=true , address=145.79.227.44:4203 , piece_id=WHCKQ2FRLYAZE7Q6HYZNTS4EISVHZNAPDDKH34WN33ALK4MVK2EA, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:14.338651537 (590h53m14.209734582s) [noise=true , address=77.90.31.15:4729 , piece_id=NVMXFEILHGT622IIPYYT4FFHC2WZLHMCZM4VT5WUPHRAPH3DX6TA, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:14.344145168 (590h53m14.215228213s) [noise=true , address=145.79.227.30:4772 , piece_id=PVMD5DWRRPVARTSZ6O7HN6MUVTZUYWLYSNQR5GDMNKUJZGXMYPFQ, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:14.667232847 (590h53m14.538315892s) [noise=true , address=193.149.164.143:4704 , piece_id=4T2IJIA6WHI5LMIUP5BLJLCD3GRY3A6PXR6G5OWJWUSYZT5ELGHQ, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:14.671388990 (590h53m14.542472025s) [noise=true , address=145.79.227.29:4945 , piece_id=SHL73IDEI54GRCWYNWHRUSJZZ4UUXREAX54DFJLTG5QPZCF5QKIQ, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:14.671415190 (590h53m14.542498226s) [seg_pos={0 0}, upload=517998]: Segment reached optimal threshold of 14 pieces.
Dec 30 06:22:14.671803600 (590h53m14.542886655s) [noise=true , address=145.79.227.40:4070 , piece_id=YOAGDXNN4TTCK35LLQDSP24BOWLZXGNSQU7WMKEEVVMPXDVI74JA, seg_pos={0 0}, upload=517998]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12hU1LwL2gX7rVnrNgzaBCYB37ziqf7s7aUgtF2Uc3antReQ6me): protocol: expected piece hash; context canceled; EOF
Dec 30 06:22:14.671814470 (590h53m14.542897525s) [noise=true , address=145.79.227.40:4070 , piece_id=YOAGDXNN4TTCK35LLQDSP24BOWLZXGNSQU7WMKEEVVMPXDVI74JA, seg_pos={0 0}, upload=517998]: Long tail cancellation due to optimal threshold
Dec 30 06:22:14.672014932 (590h53m14.543097977s) [noise=true , address=145.79.227.43:4177 , piece_id=YM37ZI3WNZ7SXGTFC224TOVVTHYNFIPRQB2TFLEZ735RNZE553FQ, seg_pos={0 0}, upload=517998]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bhN3cR22QuAYihnnB3x3R97UZRWJ8sFPP8fjQcXZ7qhNZphZG): protocol: expected piece hash; context canceled; EOF
Dec 30 06:22:14.672022637 (590h53m14.543105672s) [noise=true , address=145.79.227.43:4177 , piece_id=YM37ZI3WNZ7SXGTFC224TOVVTHYNFIPRQB2TFLEZ735RNZE553FQ, seg_pos={0 0}, upload=517998]: Long tail cancellation due to optimal threshold
Dec 30 06:22:14.672088933 (590h53m14.543171988s) [noise=true , address=145.79.227.33:4881 , piece_id=EX66JF7U7A4ZSRWJMP4MCLQRPW2ZZFZFIUNV72XZXU7E5O3KWX3A, seg_pos={0 0}, upload=517998]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mMBjg1b4MqTZPE5NpNNMYDzmsHqWYotE5jBNmZLyA5FyYLZRp): protocol: expected piece hash; context canceled; EOF
Dec 30 06:22:14.672127677 (590h53m14.543210742s) [noise=true , address=145.79.227.33:4881 , piece_id=EX66JF7U7A4ZSRWJMP4MCLQRPW2ZZFZFIUNV72XZXU7E5O3KWX3A, seg_pos={0 0}, upload=517998]: Long tail cancellation due to optimal threshold
Dec 30 06:22:14.672312739 (590h53m14.543395774s) [noise=true , address=45.91.116.14:4217 , piece_id=L4MAHTGKTNWBSN3YTB74E3DBJWVMA3DPU57MQK5FNK35DXRZ6AIA, seg_pos={0 0}, upload=517998]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xSXWFiq27Dd2vBDD9vVnhDHVG59UmDKn3hVAhf3oyo5goVNaE): protocol: expected piece hash; context canceled; EOF
Dec 30 06:22:14.672319953 (590h53m14.543402988s) [noise=true , address=45.91.116.14:4217 , piece_id=L4MAHTGKTNWBSN3YTB74E3DBJWVMA3DPU57MQK5FNK35DXRZ6AIA, seg_pos={0 0}, upload=517998]: Long tail cancellation due to optimal threshold
Dec 30 06:22:14.672337847 (590h53m14.543420882s) [seg_pos={0 0}, upload=517998]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 30 06:22:14.672452947 (590h53m14.543535982s) [upload=517998]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 06:22:14.672458507 (590h53m14.543541542s) [upload=517998]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 06:22:14.981437069 (590h53m14.852520114s) [upload=517998]: Done uploading segments...
Dec 30 06:22:14.981456546 (590h53m14.852539581s) [upload=517998]: Upload finished. err: <nil>
Duration: 40.778900652s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [..............X.............................................] duration:
8.570845881s trace:
https://jaeger.dev.storj.io/trace/deb96436003dee
FAIL (2025-12-30 06:47:06 UTC)
orbiter warning: read tcp 188.166.225.170:52458->45.38.24.2:443: read: connection reset by peer
Duration: 8.570870046s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
14 out of 60 is failed (threshold=3) [..X..X......X....X...X..X.X.X.....X.......X.X.....X....XX...] duration:
5.903364584s trace:
https://jaeger.dev.storj.io/trace/78056e8d4b3848bb
FAIL (2025-12-30 06:46:42 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
Dec 30 06:43:43.438454167 (591h14m43.309537212s) [upload=518325]: Starting upload
Dec 30 06:43:43.438541793 (591h14m43.309624849s) [upload=518325]: Done starting upload
Dec 30 06:43:43.438670769 (591h14m43.309753824s) [upload=518325]: Uploading segments...
Dec 30 06:43:43.438849339 (591h14m43.309932394s) [upload=518325]: Got next segment. Inline: false
Dec 30 06:43:43.438883805 (591h14m43.30996685s) [upload=518325]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 30 06:43:43.438889526 (591h14m43.309972561s) [upload=518325]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 30 06:43:43.682039201 (591h14m43.553122246s) [seg_pos={0 0}, upload=518325]: Begin upload segment...
Dec 30 06:43:43.682106038 (591h14m43.553189083s) [seg_pos={0 0}, upload=518325]: Done begin upload segment.
Dec 30 06:43:43.682152857 (591h14m43.553235902s) [noise=true , address=145.79.227.11:4732 , piece_id=WOMYTSZBK7KZRKDYYZA3ORJ5M6RCYJUHU2O7GXYY3CSQVFYE3S4A, seg_pos={0 0}, upload=518325]: Uploading piece...
Dec 30 06:43:43.682258869 (591h14m43.553341914s) [noise=true , address=77.90.31.17:4770 , piece_id=EZYUKNUOTNWRXEROZAVMPDNQ2EAYUBO3KP7X2GWSQXU6ZRQ5EJQQ, seg_pos={0 0}, upload=518325]: Uploading piece...
Dec 30 06:43:43.683367860 (591h14m43.554450906s) [noise=true , address=145.79.227.14:4838 , piece_id=LNTWJQBCQVT7WXSCH5EIII7DHI4SVOYIVKK4F7IXIKM4KMED2YMQ, seg_pos={0 0}, upload=518325]: Uploading piece...
Dec 30 06:43:43.683377639 (591h14m43.554460684s) [noise=true , address=145.79.227.36:4518 , piece_id=KWGCY63F2DZC5MNJIY5T73RQHOGDUNIOHOMFBXJXWIQXFQA73PYA, seg_pos={0 0}, upload=518325]: Uploading piece...
Dec 30 06:43:43.683832937 (591h14m43.554915972s) [noise=true , address=45.91.116.11:4120 , piece_id=VTJS4ZRPH6KEIEAYEQ4ODRTTGTX6BA2FWMZRTNT75OF7SH2DZUDA, seg_pos={0 0}, upload=518325]: Uploading piece...
Dec 30 06:43:43.684783196 (591h14m43.555866241s) [noise=true , address=145.79.227.18:4007 , piece_id=OWWYNA55RHUTUGUQT4Y2ZBI3VQAQQVNJTFKG6ALMRXXDLW3DW4AQ, seg_pos={0 0}, upload=518325]: Uploading piece...
Dec 30 06:43:43.685038492 (591h14m43.556121527s) [noise=true , address=145.79.227.3:4151 , piece_id=EJ6XIYN4PQ5KGO6EYP4YMQHXEZKZ2NHASAJTNVQ3VF7S5PNHCYKA, seg_pos={0 0}, upload=518325]: Uploading piece...
Dec 30 06:43:43.686167853 (591h14m43.557250908s) [noise=true , address=45.91.116.9:4263 , piece_id=AS5TS3VJSLQMF7ZHLF7B2VBTDVIB23LHUU63EGI2ENDXGEYPDV5A, seg_pos={0 0}, upload=518325]: Uploading piece...
Dec 30 06:43:43.686258695 (591h14m43.55734174s) [noise=true , address=145.79.227.29:4943 , piece_id=TDSZ3VSCCNAU73ZL2CS5XPWOCA6TRTZZKY7VDS2WSK3E2PFFTNMA, seg_pos={0 0}, upload=518325]: Uploading piece...
Dec 30 06:43:43.687362998 (591h14m43.558446043s) [noise=true , address=45.91.116.10:4083 , piece_id=AFTCCUF3L7226XLTU4GEFEKYGYQAQPJKHIPKX5N7MT2UCU45L2BQ, seg_pos={0 0}, upload=518325]: Uploading piece...
Dec 30 06:43:43.687636299 (591h14m43.558719334s) [noise=true , address=145.79.227.50:4374 , piece_id=W3NSQQDQ72JDIVU3LX7BQSWTCTEM3UAZLH2GU2AM7I7DWIRHFKZA, seg_pos={0 0}, upload=518325]: Uploading piece...
Dec 30 06:43:43.688312987 (591h14m43.559396022s) [noise=true , address=145.79.227.26:4654 , piece_id=J7RRD6V3GTKDEZEPPOUNPBAMD3Q4QHIG7T6Z7FE72EJAJKSDTGBQ, seg_pos={0 0}, upload=518325]: Uploading piece...
Dec 30 06:43:43.688619340 (591h14m43.559702385s) [noise=true , address=77.90.31.15:4697 , piece_id=ZQJMG5N6Z5DYDTEMF5EL5YQGQVGKPEWMG6RZN6EXMEJYXJUQT3WA, seg_pos={0 0}, upload=518325]: Uploading piece...
Dec 30 06:43:43.688729590 (591h14m43.559812635s) [noise=true , address=145.79.227.6:4257 , piece_id=IMXIK3WJDAMAF7QXSYTWHGLDHUGQIBXQ3WDIZX45L4ODC35H7ICQ, seg_pos={0 0}, upload=518325]: Uploading piece...
Dec 30 06:43:43.689649502 (591h14m43.560732537s) [noise=true , address=77.90.31.10:4517 , piece_id=JXYAH2NNMGP5LSLNUGVUCG3X5NDPBW3MQ6ISE6HV4XPY7Y2CKSNA, seg_pos={0 0}, upload=518325]: Uploading piece...
Dec 30 06:43:43.690303357 (591h14m43.561386402s) [noise=true , address=145.79.227.51:4410 , piece_id=HDGHD2ORSRHHSIJG23HYPYM7GATOSAOD24MYETBMKCICOSTFCJMA, seg_pos={0 0}, upload=518325]: Uploading piece...
Dec 30 06:43:45.296790318 (591h14m45.167873363s) [upload=518325]: Next returned nil segment
Dec 30 06:43:45.296952557 (591h14m45.168035602s) [upload=518325]: Waiting for error group managing segments...
Dec 30 06:43:45.672927161 (591h14m45.544010236s) [noise=true , address=145.79.227.50:4374 , piece_id=W3NSQQDQ72JDIVU3LX7BQSWTCTEM3UAZLH2GU2AM7I7DWIRHFKZA, seg_pos={0 0}, upload=518325]: Done uploading piece. err: <nil>
Dec 30 06:43:45.679981644 (591h14m45.551064679s) [noise=true , address=145.79.227.14:4838 , piece_id=LNTWJQBCQVT7WXSCH5EIII7DHI4SVOYIVKK4F7IXIKM4KMED2YMQ, seg_pos={0 0}, upload=518325]: Done uploading piece. err: <nil>
Dec 30 06:43:45.681994216 (591h14m45.553077251s) [noise=true , address=145.79.227.11:4732 , piece_id=WOMYTSZBK7KZRKDYYZA3ORJ5M6RCYJUHU2O7GXYY3CSQVFYE3S4A, seg_pos={0 0}, upload=518325]: Done uploading piece. err: <nil>
Dec 30 06:43:45.682769572 (591h14m45.553852607s) [noise=true , address=145.79.227.36:4518 , piece_id=KWGCY63F2DZC5MNJIY5T73RQHOGDUNIOHOMFBXJXWIQXFQA73PYA, seg_pos={0 0}, upload=518325]: Done uploading piece. err: <nil>
Dec 30 06:43:45.685368261 (591h14m45.556451296s) [noise=true , address=145.79.227.51:4410 , piece_id=HDGHD2ORSRHHSIJG23HYPYM7GATOSAOD24MYETBMKCICOSTFCJMA, seg_pos={0 0}, upload=518325]: Done uploading piece. err: <nil>
Dec 30 06:43:45.696508394 (591h14m45.567591429s) [noise=true , address=145.79.227.26:4654 , piece_id=J7RRD6V3GTKDEZEPPOUNPBAMD3Q4QHIG7T6Z7FE72EJAJKSDTGBQ, seg_pos={0 0}, upload=518325]: Done uploading piece. err: <nil>
Dec 30 06:43:45.698096838 (591h14m45.569179883s) [noise=true , address=145.79.227.6:4257 , piece_id=IMXIK3WJDAMAF7QXSYTWHGLDHUGQIBXQ3WDIZX45L4ODC35H7ICQ, seg_pos={0 0}, upload=518325]: Done uploading piece. err: <nil>
Dec 30 06:43:45.699199428 (591h14m45.570282463s) [noise=true , address=145.79.227.18:4007 , piece_id=OWWYNA55RHUTUGUQT4Y2ZBI3VQAQQVNJTFKG6ALMRXXDLW3DW4AQ, seg_pos={0 0}, upload=518325]: Done uploading piece. err: <nil>
Dec 30 06:43:45.758676632 (591h14m45.629759677s) [noise=true , address=145.79.227.3:4151 , piece_id=EJ6XIYN4PQ5KGO6EYP4YMQHXEZKZ2NHASAJTNVQ3VF7S5PNHCYKA, seg_pos={0 0}, upload=518325]: Done uploading piece. err: <nil>
Dec 30 06:43:45.903122208 (591h14m45.774205253s) [noise=true , address=77.90.31.10:4517 , piece_id=JXYAH2NNMGP5LSLNUGVUCG3X5NDPBW3MQ6ISE6HV4XPY7Y2CKSNA, seg_pos={0 0}, upload=518325]: Done uploading piece. err: <nil>
Dec 30 06:43:45.976028733 (591h14m45.847111778s) [noise=true , address=145.79.227.29:4943 , piece_id=TDSZ3VSCCNAU73ZL2CS5XPWOCA6TRTZZKY7VDS2WSK3E2PFFTNMA, seg_pos={0 0}, upload=518325]: Done uploading piece. err: <nil>
Dec 30 06:43:45.982909666 (591h14m45.853992711s) [noise=true , address=77.90.31.17:4770 , piece_id=EZYUKNUOTNWRXEROZAVMPDNQ2EAYUBO3KP7X2GWSQXU6ZRQ5EJQQ, seg_pos={0 0}, upload=518325]: Done uploading piece. err: <nil>
Dec 30 06:43:46.039347542 (591h14m45.910430587s) [noise=true , address=45.91.116.10:4083 , piece_id=AFTCCUF3L7226XLTU4GEFEKYGYQAQPJKHIPKX5N7MT2UCU45L2BQ, seg_pos={0 0}, upload=518325]: Done uploading piece. err: <nil>
Dec 30 06:43:46.039372420 (591h14m45.910455475s) [seg_pos={0 0}, upload=518325]: Segment reached optimal threshold of 13 pieces.
Dec 30 06:43:46.039598750 (591h14m45.910681805s) [noise=true , address=77.90.31.15:4697 , piece_id=ZQJMG5N6Z5DYDTEMF5EL5YQGQVGKPEWMG6RZN6EXMEJYXJUQT3WA, seg_pos={0 0}, upload=518325]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5): protocol: expected piece hash; context canceled; EOF
Dec 30 06:43:46.039606464 (591h14m45.91068951s) [noise=true , address=77.90.31.15:4697 , piece_id=ZQJMG5N6Z5DYDTEMF5EL5YQGQVGKPEWMG6RZN6EXMEJYXJUQT3WA, seg_pos={0 0}, upload=518325]: Long tail cancellation due to optimal threshold
Dec 30 06:43:46.039875217 (591h14m45.910958282s) [noise=true , address=45.91.116.9:4263 , piece_id=AS5TS3VJSLQMF7ZHLF7B2VBTDVIB23LHUU63EGI2ENDXGEYPDV5A, seg_pos={0 0}, upload=518325]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF): protocol: expected piece hash; context canceled; EOF
Dec 30 06:43:46.039883282 (591h14m45.910966327s) [noise=true , address=45.91.116.9:4263 , piece_id=AS5TS3VJSLQMF7ZHLF7B2VBTDVIB23LHUU63EGI2ENDXGEYPDV5A, seg_pos={0 0}, upload=518325]: Long tail cancellation due to optimal threshold
Dec 30 06:43:46.039942615 (591h14m45.91102568s) [noise=true , address=45.91.116.11:4120 , piece_id=VTJS4ZRPH6KEIEAYEQ4ODRTTGTX6BA2FWMZRTNT75OF7SH2DZUDA, seg_pos={0 0}, upload=518325]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8): protocol: expected piece hash; context canceled; EOF
Dec 30 06:43:46.039962774 (591h14m45.911045829s) [noise=true , address=45.91.116.11:4120 , piece_id=VTJS4ZRPH6KEIEAYEQ4ODRTTGTX6BA2FWMZRTNT75OF7SH2DZUDA, seg_pos={0 0}, upload=518325]: Long tail cancellation due to optimal threshold
Dec 30 06:43:46.039995456 (591h14m45.911078511s) [seg_pos={0 0}, upload=518325]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 30 06:43:46.040216396 (591h14m45.911299441s) [upload=518325]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 06:43:46.040225283 (591h14m45.911308328s) [upload=518325]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 06:43:46.331063484 (591h14m46.202146549s) [upload=518325]: Done uploading segments...
Dec 30 06:43:46.331089314 (591h14m46.202172349s) [upload=518325]: Upload finished. err: <nil>
Duration: 5.903392748s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [...............X............................................] duration:
7.009145763s trace:
https://jaeger.dev.storj.io/trace/deb96436003dee
FAIL (2025-12-30 06:46:39 UTC)
orbiter warning: read tcp 188.166.225.170:52458->45.38.24.2:443: read: connection reset by peer
Duration: 7.009167364s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)antiblip:
1 out of 60 is failed (threshold=3) [.....................................................X......] duration:
7.192163033s trace:
https://jaeger.dev.storj.io/trace/4c2f4efaff0e0ea0
FAIL (2025-12-30 06:46:22 UTC)
orbiter warning:
operation: DeleteObject
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
load=518292]: Uploading piece...
Dec 30 06:41:13.598100152 (591h12m13.469183187s) [noise=true , address=81.171.14.70:4041 , piece_id=NORNA6WFTGPNOHU7MPTNNB5ONTBS5DJGCN2NWIZAEPYSUSIOSGEQ, seg_pos={0 0}, upload=518292]: Uploading piece...
Dec 30 06:41:13.598587971 (591h12m13.469671016s) [noise=true , address=89.149.201.147:4007 , piece_id=HZLSYGSBP4TJH5V5XF3Q3AETJU625ERNDBD3VXW2HUZTTPAHLBMA, seg_pos={0 0}, upload=518292]: Uploading piece...
Dec 30 06:41:13.599093143 (591h12m13.470176178s) [noise=true , address=89.149.201.147:4008 , piece_id=XYFRNJVIFFZA5S36KFPGZPVCPLRL2OW5QBJD3UR37VPPA2RX3RJA, seg_pos={0 0}, upload=518292]: Uploading piece...
Dec 30 06:41:13.599699999 (591h12m13.470783034s) [noise=true , address=81.171.14.70:4039 , piece_id=DEJDXFDHOSAMIP2FN5LJBJNBB4QLJSE2AUDW6IMA6IC4VRY7VAGQ, seg_pos={0 0}, upload=518292]: Uploading piece...
Dec 30 06:41:13.600136169 (591h12m13.471219204s) [noise=true , address=185.167.204.205:4034 , piece_id=OWZHQB5ZDT2QZ2QAHZJVRWQ222MX6CWLKJUOIAAGSMAVLWGJAOBA, seg_pos={0 0}, upload=518292]: Uploading piece...
Dec 30 06:41:13.600543184 (591h12m13.471626229s) [noise=true , address=185.167.204.206:4058 , piece_id=ROOCGK64M5RHIWNAVCSZNYTZ6M2ORWB7MJ27XJVDTFVQIKJZHY4A, seg_pos={0 0}, upload=518292]: Uploading piece...
Dec 30 06:41:13.601048286 (591h12m13.472131332s) [noise=true , address=81.171.14.70:4046 , piece_id=B2MTUNKMUP5S7XW4SVQ5QI2ZO3VFB5QVAKBBY4BIZIUILM6O7VYQ, seg_pos={0 0}, upload=518292]: Uploading piece...
Dec 30 06:41:13.601698784 (591h12m13.472781819s) [noise=true , address=185.167.204.206:4064 , piece_id=H4PP7HPHXRX2LOZ4776AMWAJDN7OC435E2K4BXI7XOZYKTB4YUNA, seg_pos={0 0}, upload=518292]: Uploading piece...
Dec 30 06:41:13.602168259 (591h12m13.473251304s) [noise=true , address=185.167.204.206:4065 , piece_id=LD3J4656Q5QEUCMJ5OKEYATEVRIKSXO7XGARTN25BVM7FDIYOWKA, seg_pos={0 0}, upload=518292]: Uploading piece...
Dec 30 06:41:13.602782708 (591h12m13.473865753s) [noise=true , address=81.171.14.70:4027 , piece_id=6WEORWX5QOMFSJUEADIIBHHIPKNFP4NJPZAJADBUTPUSG7MZJMOQ, seg_pos={0 0}, upload=518292]: Uploading piece...
Dec 30 06:41:14.423395002 (591h12m14.294478047s) [upload=518292]: Next returned nil segment
Dec 30 06:41:14.423414068 (591h12m14.294497103s) [upload=518292]: Waiting for error group managing segments...
Dec 30 06:41:14.817650316 (591h12m14.688733351s) [noise=true , address=185.167.204.205:4029 , piece_id=KUOMMYJ2F4NIK6VN4C5M3Y3TMHYEZH7QFNJ6K3CLOAVI5R6VKTHQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:14.817870054 (591h12m14.688953099s) [noise=true , address=185.167.204.205:4019 , piece_id=H6AH654VGX6NHBUKODIWGL7I2RKE3DDMY4UM6VDONP2EP3NPDJBA, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:14.818623698 (591h12m14.689706744s) [noise=true , address=185.167.204.206:4065 , piece_id=LD3J4656Q5QEUCMJ5OKEYATEVRIKSXO7XGARTN25BVM7FDIYOWKA, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:14.818932436 (591h12m14.690015481s) [noise=true , address=185.167.204.206:4061 , piece_id=T2XS4IFPQSVS3T45SPQ5ZDDVT2VMTMXHKL2ZMX4OIOPRR67SCD2Q, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:14.819147046 (591h12m14.690230091s) [noise=true , address=185.167.204.206:4071 , piece_id=QI2TOY5KCDZVWCCLMAJXVUWD2HB7S7IOTEAV2EHAMUNHKOFKTNYA, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:14.819678578 (591h12m14.690761623s) [noise=true , address=185.167.204.206:4059 , piece_id=LMOGTWX2YIRR3EKNROK43O2B27S5L5U6GYB7HBVRBPQPH4WDDWZQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:14.819905028 (591h12m14.690988063s) [noise=true , address=185.167.204.206:4058 , piece_id=ROOCGK64M5RHIWNAVCSZNYTZ6M2ORWB7MJ27XJVDTFVQIKJZHY4A, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:14.820479312 (591h12m14.691562357s) [noise=true , address=185.167.204.205:4021 , piece_id=KP7J5XSY4SKSKZG7XXFZMNKMRP4472PXI7CW2UQBZICN4MAFO5RQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:14.823149296 (591h12m14.694232331s) [noise=true , address=185.167.204.205:4034 , piece_id=OWZHQB5ZDT2QZ2QAHZJVRWQ222MX6CWLKJUOIAAGSMAVLWGJAOBA, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:14.827906996 (591h12m14.698990041s) [noise=true , address=185.167.204.206:4055 , piece_id=VYEDR6XHZBZMTVJ5U5OF67WZOGVMCMQLXXBQUI7DHBCCVQ2RRERQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:14.828003769 (591h12m14.699086814s) [noise=true , address=185.167.204.205:4016 , piece_id=I6MUDLAXUUDJL2C4HFE22C2L56A7D6HSCNWXKWJ7OKO2N4RA4W7Q, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:14.828893203 (591h12m14.699976238s) [noise=true , address=185.167.204.206:4064 , piece_id=H4PP7HPHXRX2LOZ4776AMWAJDN7OC435E2K4BXI7XOZYKTB4YUNA, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:14.829477074 (591h12m14.700560109s) [noise=true , address=185.167.204.205:4017 , piece_id=CZP65KETMGOHFDMV7KENRL4V36UHBPWRDCBBPUM5NE5L2PV4AQCQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:14.829774972 (591h12m14.700858007s) [noise=true , address=185.167.204.206:4043 , piece_id=G7VZYID4TXGNPDN63NQULXOVSX42WMFFNPBX6KQJ5L2WSW4D2YGQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:14.831734493 (591h12m14.702817528s) [noise=true , address=185.167.204.206:4049 , piece_id=BQ7JEGSGPIFJUGZFBYRPDMBNPZ24BG6TEG4VUM4JUBTE4T2RF4SA, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:15.405688561 (591h12m15.276771606s) [noise=true , address=81.171.14.70:4027 , piece_id=6WEORWX5QOMFSJUEADIIBHHIPKNFP4NJPZAJADBUTPUSG7MZJMOQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:15.417110440 (591h12m15.288193485s) [noise=true , address=81.171.14.70:4031 , piece_id=U6Z4SNK32ZSWQ7NDR7ACXGRC4BADDOPR65ILBR64QLGVNDBADFFQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:15.425600446 (591h12m15.296683481s) [noise=true , address=81.171.14.70:4037 , piece_id=6A25LEIAQEJB3DNCTXQT6K7ZSKIGOW4QUGMNKLOG3CT2XXH25PSA, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:15.499056228 (591h12m15.370139263s) [noise=true , address=81.171.14.70:4038 , piece_id=3WRHW3VGXWHOEOV5Q6CLYSF7MUVWS7QHTNHLSTCKAUERPHAH3I3Q, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:15.514998925 (591h12m15.38608197s) [noise=true , address=81.171.14.70:4041 , piece_id=NORNA6WFTGPNOHU7MPTNNB5ONTBS5DJGCN2NWIZAEPYSUSIOSGEQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:15.528579544 (591h12m15.399662589s) [noise=true , address=81.171.14.70:4030 , piece_id=KITOJIT34CHS2QZBXCW2SJPZB4RHEHCMJ6IYQ4Z5KLL24YAWBIUQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:15.528991398 (591h12m15.400074434s) [noise=true , address=81.171.14.70:4047 , piece_id=ZWV5DP4YC3MSYDWEQH4ZK3VR4WZ4DUTI4VL3WVLQKFIEKIW44LOQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:15.534844544 (591h12m15.405927579s) [noise=true , address=81.171.14.70:4035 , piece_id=XCLL5SOAPJSSGF5YYQH6ZKGOHMLS6RI6H26FOW5XIJRY3TWYNEGA, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:15.915614910 (591h12m15.786697945s) [noise=true , address=81.171.14.70:4039 , piece_id=DEJDXFDHOSAMIP2FN5LJBJNBB4QLJSE2AUDW6IMA6IC4VRY7VAGQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:15.967820176 (591h12m15.838903211s) [noise=true , address=81.171.14.70:4046 , piece_id=B2MTUNKMUP5S7XW4SVQ5QI2ZO3VFB5QVAKBBY4BIZIUILM6O7VYQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:16.172116128 (591h12m16.043199163s) [noise=true , address=89.149.201.147:4007 , piece_id=HZLSYGSBP4TJH5V5XF3Q3AETJU625ERNDBD3VXW2HUZTTPAHLBMA, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:16.172137559 (591h12m16.043220604s) [seg_pos={0 0}, upload=518292]: Segment reached optimal threshold of 26 pieces.
Dec 30 06:41:16.172336949 (591h12m16.043419994s) [noise=true , address=89.149.201.147:4008 , piece_id=XYFRNJVIFFZA5S36KFPGZPVCPLRL2OW5QBJD3UR37VPPA2RX3RJA, seg_pos={0 0}, upload=518292]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QN8VP6o7qGw9Dd8EafU9xe7s7Cuf1FyqPMbnRaznrfPxfEUUn): protocol: expected piece hash; context canceled; EOF
Dec 30 06:41:16.172344213 (591h12m16.043427258s) [noise=true , address=89.149.201.147:4008 , piece_id=XYFRNJVIFFZA5S36KFPGZPVCPLRL2OW5QBJD3UR37VPPA2RX3RJA, seg_pos={0 0}, upload=518292]: Long tail cancellation due to optimal threshold
Dec 30 06:41:16.172554002 (591h12m16.043637037s) [noise=true , address=89.149.201.147:4020 , piece_id=N7G5DGEDYVKBTO5S6BYW3IYEOK7HAMK5JQIBMVPMPYRDHXODWEGA, seg_pos={0 0}, upload=518292]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12YeKHJH7CoouLFJYENJhzVLNkA32hJaqjLz3dXhr78eaSGtcic): protocol: expected piece hash; context canceled; EOF
Dec 30 06:41:16.172565233 (591h12m16.043648278s) [noise=true , address=89.149.201.147:4020 , piece_id=N7G5DGEDYVKBTO5S6BYW3IYEOK7HAMK5JQIBMVPMPYRDHXODWEGA, seg_pos={0 0}, upload=518292]: Long tail cancellation due to optimal threshold
Dec 30 06:41:16.172587506 (591h12m16.043670541s) [seg_pos={0 0}, upload=518292]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil>
Dec 30 06:41:16.172717945 (591h12m16.04380098s) [upload=518292]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 06:41:16.172724567 (591h12m16.043807602s) [upload=518292]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 06:41:16.404092237 (591h12m16.275175292s) [upload=518292]: Done uploading segments...
Dec 30 06:41:16.404119940 (591h12m16.275202975s) [upload=518292]: Upload finished. err: <nil>
Duration: 7.192177501s
core
uplink
eu-select-1
eu1-placement-15
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
9 out of 60 is failed (threshold=6) [...X...........X..............XX...X......X..X..X.....X.....] duration:
3.503724364s trace:
https://jaeger.dev.storj.io/trace/7d0f3c444f9c906c
FAIL (2025-12-30 06:46:19 UTC)
orbiter warning: context deadline exceeded
Duration: 3.503757175s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
4 out of 60 is failed (threshold=3) [.....X...X......X............X..............................] duration:
18.495955381s trace:
https://jaeger.dev.storj.io/trace/45b769fc71e13448
FAIL (2025-12-30 06:46:09 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: context deadline exceeded
======LOGS======
.577697683 (590h53m11.448780728s) [noise=true , address=193.149.164.143:4704 , piece_id=4T2IJIA6WHI5LMIUP5BLJLCD3GRY3A6PXR6G5OWJWUSYZT5ELGHQ, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.578173528 (590h53m11.449256573s) [noise=true , address=64.49.13.27:4965 , piece_id=CO4O5F2IBZAAKBZBV3HWNNOXZOGLNNDPALWMQZ3DEM3MREASVFEA, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.578542992 (590h53m11.449626027s) [noise=true , address=77.90.31.15:4729 , piece_id=NVMXFEILHGT622IIPYYT4FFHC2WZLHMCZM4VT5WUPHRAPH3DX6TA, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.578846911 (590h53m11.449929946s) [noise=true , address=145.79.227.43:4177 , piece_id=YM37ZI3WNZ7SXGTFC224TOVVTHYNFIPRQB2TFLEZ735RNZE553FQ, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.579458044 (590h53m11.450541089s) [noise=true , address=45.91.116.11:4150 , piece_id=E66IDHTJDLYLYJIOWMOIKOOWLXCUNCMBIQ7SQBAD3VTJF6VX5JLQ, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.579891209 (590h53m11.450974245s) [noise=true , address=145.79.227.29:4945 , piece_id=SHL73IDEI54GRCWYNWHRUSJZZ4UUXREAX54DFJLTG5QPZCF5QKIQ, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.579915535 (590h53m11.45099858s) [noise=true , address=145.79.227.30:4772 , piece_id=PVMD5DWRRPVARTSZ6O7HN6MUVTZUYWLYSNQR5GDMNKUJZGXMYPFQ, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.580325466 (590h53m11.451408511s) [noise=true , address=77.90.31.12:4603 , piece_id=2I4L7F2WEGK5OSLODJSJ4672URBA7DUGXWZKTYB6ZNA2TQIGVXWA, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.580379599 (590h53m11.451462644s) [noise=true , address=77.90.31.14:4671 , piece_id=263TRISJ5ADFTX43ULSSGBAF2KMOXAPISJ56LLE53D74XOEMQ5EA, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.580692044 (590h53m11.451775089s) [noise=true , address=145.79.227.3:4175 , piece_id=UCV5WQNVWEAMEDBWDNXEVSQEGXPJE7KD6YSWC5A4WXYUKUCK4NLA, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.580857859 (590h53m11.451940894s) [noise=true , address=77.90.31.19:4860 , piece_id=U2MNFOAD7FHCDBE4FUYOROIP3P33ROJ7K75SF5Z747EFKFTF7RGA, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.581089210 (590h53m11.452172255s) [noise=true , address=145.79.227.44:4203 , piece_id=WHCKQ2FRLYAZE7Q6HYZNTS4EISVHZNAPDDKH34WN33ALK4MVK2EA, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.581351320 (590h53m11.452434355s) [noise=true , address=64.49.13.154:4928 , piece_id=SEVBZBMNGURP5TRUEBA4PHRSA6MNANVJUKINALJUWPQWBGKVYNTA, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.581457091 (590h53m11.452540126s) [noise=true , address=145.79.227.34:4909 , piece_id=3QSWTNR5LGUUAY3N5HJKSPL3TRQ6WO5F5OD4VTXKWHWUXUF7PVVQ, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.581863144 (590h53m11.452946179s) [noise=true , address=145.79.227.33:4881 , piece_id=EX66JF7U7A4ZSRWJMP4MCLQRPW2ZZFZFIUNV72XZXU7E5O3KWX3A, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.584010443 (590h53m11.455093478s) [noise=true , address=145.79.227.40:4070 , piece_id=YOAGDXNN4TTCK35LLQDSP24BOWLZXGNSQU7WMKEEVVMPXDVI74JA, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.585122791 (590h53m11.456205826s) [noise=true , address=45.91.116.14:4217 , piece_id=L4MAHTGKTNWBSN3YTB74E3DBJWVMA3DPU57MQK5FNK35DXRZ6AIA, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:12.983305402 (590h53m12.854388447s) [upload=517998]: Next returned nil segment
Dec 30 06:22:12.983321313 (590h53m12.854404348s) [upload=517998]: Waiting for error group managing segments...
Dec 30 06:22:13.379703053 (590h53m13.250786088s) [noise=true , address=64.49.13.27:4965 , piece_id=CO4O5F2IBZAAKBZBV3HWNNOXZOGLNNDPALWMQZ3DEM3MREASVFEA, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:13.418055622 (590h53m13.289138657s) [noise=true , address=145.79.227.3:4175 , piece_id=UCV5WQNVWEAMEDBWDNXEVSQEGXPJE7KD6YSWC5A4WXYUKUCK4NLA, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:13.509034160 (590h53m13.380117205s) [noise=true , address=64.49.13.154:4928 , piece_id=SEVBZBMNGURP5TRUEBA4PHRSA6MNANVJUKINALJUWPQWBGKVYNTA, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:13.642886154 (590h53m13.513969199s) [noise=true , address=77.90.31.14:4671 , piece_id=263TRISJ5ADFTX43ULSSGBAF2KMOXAPISJ56LLE53D74XOEMQ5EA, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:13.788145520 (590h53m13.659228555s) [noise=true , address=77.90.31.12:4603 , piece_id=2I4L7F2WEGK5OSLODJSJ4672URBA7DUGXWZKTYB6ZNA2TQIGVXWA, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:13.858878757 (590h53m13.729961792s) [noise=true , address=145.79.227.49:4369 , piece_id=XUEXGNJJIJ4DUBDNUV7CKF6J5JYHSLZAEPVGEDGBUBY42BIP4XEQ, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:13.879550569 (590h53m13.750633614s) [noise=true , address=145.79.227.34:4909 , piece_id=3QSWTNR5LGUUAY3N5HJKSPL3TRQ6WO5F5OD4VTXKWHWUXUF7PVVQ, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:13.940897723 (590h53m13.811980769s) [noise=true , address=77.90.31.19:4860 , piece_id=U2MNFOAD7FHCDBE4FUYOROIP3P33ROJ7K75SF5Z747EFKFTF7RGA, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:13.989791936 (590h53m13.860874981s) [noise=true , address=45.91.116.11:4150 , piece_id=E66IDHTJDLYLYJIOWMOIKOOWLXCUNCMBIQ7SQBAD3VTJF6VX5JLQ, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:14.289516718 (590h53m14.160599753s) [noise=true , address=145.79.227.44:4203 , piece_id=WHCKQ2FRLYAZE7Q6HYZNTS4EISVHZNAPDDKH34WN33ALK4MVK2EA, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:14.338651537 (590h53m14.209734582s) [noise=true , address=77.90.31.15:4729 , piece_id=NVMXFEILHGT622IIPYYT4FFHC2WZLHMCZM4VT5WUPHRAPH3DX6TA, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:14.344145168 (590h53m14.215228213s) [noise=true , address=145.79.227.30:4772 , piece_id=PVMD5DWRRPVARTSZ6O7HN6MUVTZUYWLYSNQR5GDMNKUJZGXMYPFQ, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:14.667232847 (590h53m14.538315892s) [noise=true , address=193.149.164.143:4704 , piece_id=4T2IJIA6WHI5LMIUP5BLJLCD3GRY3A6PXR6G5OWJWUSYZT5ELGHQ, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:14.671388990 (590h53m14.542472025s) [noise=true , address=145.79.227.29:4945 , piece_id=SHL73IDEI54GRCWYNWHRUSJZZ4UUXREAX54DFJLTG5QPZCF5QKIQ, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:14.671415190 (590h53m14.542498226s) [seg_pos={0 0}, upload=517998]: Segment reached optimal threshold of 14 pieces.
Dec 30 06:22:14.671803600 (590h53m14.542886655s) [noise=true , address=145.79.227.40:4070 , piece_id=YOAGDXNN4TTCK35LLQDSP24BOWLZXGNSQU7WMKEEVVMPXDVI74JA, seg_pos={0 0}, upload=517998]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12hU1LwL2gX7rVnrNgzaBCYB37ziqf7s7aUgtF2Uc3antReQ6me): protocol: expected piece hash; context canceled; EOF
Dec 30 06:22:14.671814470 (590h53m14.542897525s) [noise=true , address=145.79.227.40:4070 , piece_id=YOAGDXNN4TTCK35LLQDSP24BOWLZXGNSQU7WMKEEVVMPXDVI74JA, seg_pos={0 0}, upload=517998]: Long tail cancellation due to optimal threshold
Dec 30 06:22:14.672014932 (590h53m14.543097977s) [noise=true , address=145.79.227.43:4177 , piece_id=YM37ZI3WNZ7SXGTFC224TOVVTHYNFIPRQB2TFLEZ735RNZE553FQ, seg_pos={0 0}, upload=517998]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bhN3cR22QuAYihnnB3x3R97UZRWJ8sFPP8fjQcXZ7qhNZphZG): protocol: expected piece hash; context canceled; EOF
Dec 30 06:22:14.672022637 (590h53m14.543105672s) [noise=true , address=145.79.227.43:4177 , piece_id=YM37ZI3WNZ7SXGTFC224TOVVTHYNFIPRQB2TFLEZ735RNZE553FQ, seg_pos={0 0}, upload=517998]: Long tail cancellation due to optimal threshold
Dec 30 06:22:14.672088933 (590h53m14.543171988s) [noise=true , address=145.79.227.33:4881 , piece_id=EX66JF7U7A4ZSRWJMP4MCLQRPW2ZZFZFIUNV72XZXU7E5O3KWX3A, seg_pos={0 0}, upload=517998]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mMBjg1b4MqTZPE5NpNNMYDzmsHqWYotE5jBNmZLyA5FyYLZRp): protocol: expected piece hash; context canceled; EOF
Dec 30 06:22:14.672127677 (590h53m14.543210742s) [noise=true , address=145.79.227.33:4881 , piece_id=EX66JF7U7A4ZSRWJMP4MCLQRPW2ZZFZFIUNV72XZXU7E5O3KWX3A, seg_pos={0 0}, upload=517998]: Long tail cancellation due to optimal threshold
Dec 30 06:22:14.672312739 (590h53m14.543395774s) [noise=true , address=45.91.116.14:4217 , piece_id=L4MAHTGKTNWBSN3YTB74E3DBJWVMA3DPU57MQK5FNK35DXRZ6AIA, seg_pos={0 0}, upload=517998]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xSXWFiq27Dd2vBDD9vVnhDHVG59UmDKn3hVAhf3oyo5goVNaE): protocol: expected piece hash; context canceled; EOF
Dec 30 06:22:14.672319953 (590h53m14.543402988s) [noise=true , address=45.91.116.14:4217 , piece_id=L4MAHTGKTNWBSN3YTB74E3DBJWVMA3DPU57MQK5FNK35DXRZ6AIA, seg_pos={0 0}, upload=517998]: Long tail cancellation due to optimal threshold
Dec 30 06:22:14.672337847 (590h53m14.543420882s) [seg_pos={0 0}, upload=517998]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 30 06:22:14.672452947 (590h53m14.543535982s) [upload=517998]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 06:22:14.672458507 (590h53m14.543541542s) [upload=517998]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 06:22:14.981437069 (590h53m14.852520114s) [upload=517998]: Done uploading segments...
Dec 30 06:22:14.981456546 (590h53m14.852539581s) [upload=517998]: Upload finished. err: <nil>
Duration: 18.495983837s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
14 out of 60 is failed (threshold=3) [...X..X......X....X...X..X.X.X.....X.......X.X.....X....XX..] duration:
5.957935723s trace:
https://jaeger.dev.storj.io/trace/78056e8d4b3848bb
FAIL (2025-12-30 06:46:08 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
Dec 30 06:43:43.438454167 (591h14m43.309537212s) [upload=518325]: Starting upload
Dec 30 06:43:43.438541793 (591h14m43.309624849s) [upload=518325]: Done starting upload
Dec 30 06:43:43.438670769 (591h14m43.309753824s) [upload=518325]: Uploading segments...
Dec 30 06:43:43.438849339 (591h14m43.309932394s) [upload=518325]: Got next segment. Inline: false
Dec 30 06:43:43.438883805 (591h14m43.30996685s) [upload=518325]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 30 06:43:43.438889526 (591h14m43.309972561s) [upload=518325]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 30 06:43:43.682039201 (591h14m43.553122246s) [seg_pos={0 0}, upload=518325]: Begin upload segment...
Dec 30 06:43:43.682106038 (591h14m43.553189083s) [seg_pos={0 0}, upload=518325]: Done begin upload segment.
Dec 30 06:43:43.682152857 (591h14m43.553235902s) [noise=true , address=145.79.227.11:4732 , piece_id=WOMYTSZBK7KZRKDYYZA3ORJ5M6RCYJUHU2O7GXYY3CSQVFYE3S4A, seg_pos={0 0}, upload=518325]: Uploading piece...
Dec 30 06:43:43.682258869 (591h14m43.553341914s) [noise=true , address=77.90.31.17:4770 , piece_id=EZYUKNUOTNWRXEROZAVMPDNQ2EAYUBO3KP7X2GWSQXU6ZRQ5EJQQ, seg_pos={0 0}, upload=518325]: Uploading piece...
Dec 30 06:43:43.683367860 (591h14m43.554450906s) [noise=true , address=145.79.227.14:4838 , piece_id=LNTWJQBCQVT7WXSCH5EIII7DHI4SVOYIVKK4F7IXIKM4KMED2YMQ, seg_pos={0 0}, upload=518325]: Uploading piece...
Dec 30 06:43:43.683377639 (591h14m43.554460684s) [noise=true , address=145.79.227.36:4518 , piece_id=KWGCY63F2DZC5MNJIY5T73RQHOGDUNIOHOMFBXJXWIQXFQA73PYA, seg_pos={0 0}, upload=518325]: Uploading piece...
Dec 30 06:43:43.683832937 (591h14m43.554915972s) [noise=true , address=45.91.116.11:4120 , piece_id=VTJS4ZRPH6KEIEAYEQ4ODRTTGTX6BA2FWMZRTNT75OF7SH2DZUDA, seg_pos={0 0}, upload=518325]: Uploading piece...
Dec 30 06:43:43.684783196 (591h14m43.555866241s) [noise=true , address=145.79.227.18:4007 , piece_id=OWWYNA55RHUTUGUQT4Y2ZBI3VQAQQVNJTFKG6ALMRXXDLW3DW4AQ, seg_pos={0 0}, upload=518325]: Uploading piece...
Dec 30 06:43:43.685038492 (591h14m43.556121527s) [noise=true , address=145.79.227.3:4151 , piece_id=EJ6XIYN4PQ5KGO6EYP4YMQHXEZKZ2NHASAJTNVQ3VF7S5PNHCYKA, seg_pos={0 0}, upload=518325]: Uploading piece...
Dec 30 06:43:43.686167853 (591h14m43.557250908s) [noise=true , address=45.91.116.9:4263 , piece_id=AS5TS3VJSLQMF7ZHLF7B2VBTDVIB23LHUU63EGI2ENDXGEYPDV5A, seg_pos={0 0}, upload=518325]: Uploading piece...
Dec 30 06:43:43.686258695 (591h14m43.55734174s) [noise=true , address=145.79.227.29:4943 , piece_id=TDSZ3VSCCNAU73ZL2CS5XPWOCA6TRTZZKY7VDS2WSK3E2PFFTNMA, seg_pos={0 0}, upload=518325]: Uploading piece...
Dec 30 06:43:43.687362998 (591h14m43.558446043s) [noise=true , address=45.91.116.10:4083 , piece_id=AFTCCUF3L7226XLTU4GEFEKYGYQAQPJKHIPKX5N7MT2UCU45L2BQ, seg_pos={0 0}, upload=518325]: Uploading piece...
Dec 30 06:43:43.687636299 (591h14m43.558719334s) [noise=true , address=145.79.227.50:4374 , piece_id=W3NSQQDQ72JDIVU3LX7BQSWTCTEM3UAZLH2GU2AM7I7DWIRHFKZA, seg_pos={0 0}, upload=518325]: Uploading piece...
Dec 30 06:43:43.688312987 (591h14m43.559396022s) [noise=true , address=145.79.227.26:4654 , piece_id=J7RRD6V3GTKDEZEPPOUNPBAMD3Q4QHIG7T6Z7FE72EJAJKSDTGBQ, seg_pos={0 0}, upload=518325]: Uploading piece...
Dec 30 06:43:43.688619340 (591h14m43.559702385s) [noise=true , address=77.90.31.15:4697 , piece_id=ZQJMG5N6Z5DYDTEMF5EL5YQGQVGKPEWMG6RZN6EXMEJYXJUQT3WA, seg_pos={0 0}, upload=518325]: Uploading piece...
Dec 30 06:43:43.688729590 (591h14m43.559812635s) [noise=true , address=145.79.227.6:4257 , piece_id=IMXIK3WJDAMAF7QXSYTWHGLDHUGQIBXQ3WDIZX45L4ODC35H7ICQ, seg_pos={0 0}, upload=518325]: Uploading piece...
Dec 30 06:43:43.689649502 (591h14m43.560732537s) [noise=true , address=77.90.31.10:4517 , piece_id=JXYAH2NNMGP5LSLNUGVUCG3X5NDPBW3MQ6ISE6HV4XPY7Y2CKSNA, seg_pos={0 0}, upload=518325]: Uploading piece...
Dec 30 06:43:43.690303357 (591h14m43.561386402s) [noise=true , address=145.79.227.51:4410 , piece_id=HDGHD2ORSRHHSIJG23HYPYM7GATOSAOD24MYETBMKCICOSTFCJMA, seg_pos={0 0}, upload=518325]: Uploading piece...
Dec 30 06:43:45.296790318 (591h14m45.167873363s) [upload=518325]: Next returned nil segment
Dec 30 06:43:45.296952557 (591h14m45.168035602s) [upload=518325]: Waiting for error group managing segments...
Dec 30 06:43:45.672927161 (591h14m45.544010236s) [noise=true , address=145.79.227.50:4374 , piece_id=W3NSQQDQ72JDIVU3LX7BQSWTCTEM3UAZLH2GU2AM7I7DWIRHFKZA, seg_pos={0 0}, upload=518325]: Done uploading piece. err: <nil>
Dec 30 06:43:45.679981644 (591h14m45.551064679s) [noise=true , address=145.79.227.14:4838 , piece_id=LNTWJQBCQVT7WXSCH5EIII7DHI4SVOYIVKK4F7IXIKM4KMED2YMQ, seg_pos={0 0}, upload=518325]: Done uploading piece. err: <nil>
Dec 30 06:43:45.681994216 (591h14m45.553077251s) [noise=true , address=145.79.227.11:4732 , piece_id=WOMYTSZBK7KZRKDYYZA3ORJ5M6RCYJUHU2O7GXYY3CSQVFYE3S4A, seg_pos={0 0}, upload=518325]: Done uploading piece. err: <nil>
Dec 30 06:43:45.682769572 (591h14m45.553852607s) [noise=true , address=145.79.227.36:4518 , piece_id=KWGCY63F2DZC5MNJIY5T73RQHOGDUNIOHOMFBXJXWIQXFQA73PYA, seg_pos={0 0}, upload=518325]: Done uploading piece. err: <nil>
Dec 30 06:43:45.685368261 (591h14m45.556451296s) [noise=true , address=145.79.227.51:4410 , piece_id=HDGHD2ORSRHHSIJG23HYPYM7GATOSAOD24MYETBMKCICOSTFCJMA, seg_pos={0 0}, upload=518325]: Done uploading piece. err: <nil>
Dec 30 06:43:45.696508394 (591h14m45.567591429s) [noise=true , address=145.79.227.26:4654 , piece_id=J7RRD6V3GTKDEZEPPOUNPBAMD3Q4QHIG7T6Z7FE72EJAJKSDTGBQ, seg_pos={0 0}, upload=518325]: Done uploading piece. err: <nil>
Dec 30 06:43:45.698096838 (591h14m45.569179883s) [noise=true , address=145.79.227.6:4257 , piece_id=IMXIK3WJDAMAF7QXSYTWHGLDHUGQIBXQ3WDIZX45L4ODC35H7ICQ, seg_pos={0 0}, upload=518325]: Done uploading piece. err: <nil>
Dec 30 06:43:45.699199428 (591h14m45.570282463s) [noise=true , address=145.79.227.18:4007 , piece_id=OWWYNA55RHUTUGUQT4Y2ZBI3VQAQQVNJTFKG6ALMRXXDLW3DW4AQ, seg_pos={0 0}, upload=518325]: Done uploading piece. err: <nil>
Dec 30 06:43:45.758676632 (591h14m45.629759677s) [noise=true , address=145.79.227.3:4151 , piece_id=EJ6XIYN4PQ5KGO6EYP4YMQHXEZKZ2NHASAJTNVQ3VF7S5PNHCYKA, seg_pos={0 0}, upload=518325]: Done uploading piece. err: <nil>
Dec 30 06:43:45.903122208 (591h14m45.774205253s) [noise=true , address=77.90.31.10:4517 , piece_id=JXYAH2NNMGP5LSLNUGVUCG3X5NDPBW3MQ6ISE6HV4XPY7Y2CKSNA, seg_pos={0 0}, upload=518325]: Done uploading piece. err: <nil>
Dec 30 06:43:45.976028733 (591h14m45.847111778s) [noise=true , address=145.79.227.29:4943 , piece_id=TDSZ3VSCCNAU73ZL2CS5XPWOCA6TRTZZKY7VDS2WSK3E2PFFTNMA, seg_pos={0 0}, upload=518325]: Done uploading piece. err: <nil>
Dec 30 06:43:45.982909666 (591h14m45.853992711s) [noise=true , address=77.90.31.17:4770 , piece_id=EZYUKNUOTNWRXEROZAVMPDNQ2EAYUBO3KP7X2GWSQXU6ZRQ5EJQQ, seg_pos={0 0}, upload=518325]: Done uploading piece. err: <nil>
Dec 30 06:43:46.039347542 (591h14m45.910430587s) [noise=true , address=45.91.116.10:4083 , piece_id=AFTCCUF3L7226XLTU4GEFEKYGYQAQPJKHIPKX5N7MT2UCU45L2BQ, seg_pos={0 0}, upload=518325]: Done uploading piece. err: <nil>
Dec 30 06:43:46.039372420 (591h14m45.910455475s) [seg_pos={0 0}, upload=518325]: Segment reached optimal threshold of 13 pieces.
Dec 30 06:43:46.039598750 (591h14m45.910681805s) [noise=true , address=77.90.31.15:4697 , piece_id=ZQJMG5N6Z5DYDTEMF5EL5YQGQVGKPEWMG6RZN6EXMEJYXJUQT3WA, seg_pos={0 0}, upload=518325]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5): protocol: expected piece hash; context canceled; EOF
Dec 30 06:43:46.039606464 (591h14m45.91068951s) [noise=true , address=77.90.31.15:4697 , piece_id=ZQJMG5N6Z5DYDTEMF5EL5YQGQVGKPEWMG6RZN6EXMEJYXJUQT3WA, seg_pos={0 0}, upload=518325]: Long tail cancellation due to optimal threshold
Dec 30 06:43:46.039875217 (591h14m45.910958282s) [noise=true , address=45.91.116.9:4263 , piece_id=AS5TS3VJSLQMF7ZHLF7B2VBTDVIB23LHUU63EGI2ENDXGEYPDV5A, seg_pos={0 0}, upload=518325]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF): protocol: expected piece hash; context canceled; EOF
Dec 30 06:43:46.039883282 (591h14m45.910966327s) [noise=true , address=45.91.116.9:4263 , piece_id=AS5TS3VJSLQMF7ZHLF7B2VBTDVIB23LHUU63EGI2ENDXGEYPDV5A, seg_pos={0 0}, upload=518325]: Long tail cancellation due to optimal threshold
Dec 30 06:43:46.039942615 (591h14m45.91102568s) [noise=true , address=45.91.116.11:4120 , piece_id=VTJS4ZRPH6KEIEAYEQ4ODRTTGTX6BA2FWMZRTNT75OF7SH2DZUDA, seg_pos={0 0}, upload=518325]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8): protocol: expected piece hash; context canceled; EOF
Dec 30 06:43:46.039962774 (591h14m45.911045829s) [noise=true , address=45.91.116.11:4120 , piece_id=VTJS4ZRPH6KEIEAYEQ4ODRTTGTX6BA2FWMZRTNT75OF7SH2DZUDA, seg_pos={0 0}, upload=518325]: Long tail cancellation due to optimal threshold
Dec 30 06:43:46.039995456 (591h14m45.911078511s) [seg_pos={0 0}, upload=518325]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 30 06:43:46.040216396 (591h14m45.911299441s) [upload=518325]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 06:43:46.040225283 (591h14m45.911308328s) [upload=518325]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 06:43:46.331063484 (591h14m46.202146549s) [upload=518325]: Done uploading segments...
Dec 30 06:43:46.331089314 (591h14m46.202172349s) [upload=518325]: Upload finished. err: <nil>
Duration: 5.957966311s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)antiblip:
1 out of 60 is failed (threshold=3) [......................................................X.....] duration:
6.511122231s trace:
https://jaeger.dev.storj.io/trace/4c2f4efaff0e0ea0
FAIL (2025-12-30 06:45:47 UTC)
orbiter warning:
operation: DeleteObject
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
load=518292]: Uploading piece...
Dec 30 06:41:13.598100152 (591h12m13.469183187s) [noise=true , address=81.171.14.70:4041 , piece_id=NORNA6WFTGPNOHU7MPTNNB5ONTBS5DJGCN2NWIZAEPYSUSIOSGEQ, seg_pos={0 0}, upload=518292]: Uploading piece...
Dec 30 06:41:13.598587971 (591h12m13.469671016s) [noise=true , address=89.149.201.147:4007 , piece_id=HZLSYGSBP4TJH5V5XF3Q3AETJU625ERNDBD3VXW2HUZTTPAHLBMA, seg_pos={0 0}, upload=518292]: Uploading piece...
Dec 30 06:41:13.599093143 (591h12m13.470176178s) [noise=true , address=89.149.201.147:4008 , piece_id=XYFRNJVIFFZA5S36KFPGZPVCPLRL2OW5QBJD3UR37VPPA2RX3RJA, seg_pos={0 0}, upload=518292]: Uploading piece...
Dec 30 06:41:13.599699999 (591h12m13.470783034s) [noise=true , address=81.171.14.70:4039 , piece_id=DEJDXFDHOSAMIP2FN5LJBJNBB4QLJSE2AUDW6IMA6IC4VRY7VAGQ, seg_pos={0 0}, upload=518292]: Uploading piece...
Dec 30 06:41:13.600136169 (591h12m13.471219204s) [noise=true , address=185.167.204.205:4034 , piece_id=OWZHQB5ZDT2QZ2QAHZJVRWQ222MX6CWLKJUOIAAGSMAVLWGJAOBA, seg_pos={0 0}, upload=518292]: Uploading piece...
Dec 30 06:41:13.600543184 (591h12m13.471626229s) [noise=true , address=185.167.204.206:4058 , piece_id=ROOCGK64M5RHIWNAVCSZNYTZ6M2ORWB7MJ27XJVDTFVQIKJZHY4A, seg_pos={0 0}, upload=518292]: Uploading piece...
Dec 30 06:41:13.601048286 (591h12m13.472131332s) [noise=true , address=81.171.14.70:4046 , piece_id=B2MTUNKMUP5S7XW4SVQ5QI2ZO3VFB5QVAKBBY4BIZIUILM6O7VYQ, seg_pos={0 0}, upload=518292]: Uploading piece...
Dec 30 06:41:13.601698784 (591h12m13.472781819s) [noise=true , address=185.167.204.206:4064 , piece_id=H4PP7HPHXRX2LOZ4776AMWAJDN7OC435E2K4BXI7XOZYKTB4YUNA, seg_pos={0 0}, upload=518292]: Uploading piece...
Dec 30 06:41:13.602168259 (591h12m13.473251304s) [noise=true , address=185.167.204.206:4065 , piece_id=LD3J4656Q5QEUCMJ5OKEYATEVRIKSXO7XGARTN25BVM7FDIYOWKA, seg_pos={0 0}, upload=518292]: Uploading piece...
Dec 30 06:41:13.602782708 (591h12m13.473865753s) [noise=true , address=81.171.14.70:4027 , piece_id=6WEORWX5QOMFSJUEADIIBHHIPKNFP4NJPZAJADBUTPUSG7MZJMOQ, seg_pos={0 0}, upload=518292]: Uploading piece...
Dec 30 06:41:14.423395002 (591h12m14.294478047s) [upload=518292]: Next returned nil segment
Dec 30 06:41:14.423414068 (591h12m14.294497103s) [upload=518292]: Waiting for error group managing segments...
Dec 30 06:41:14.817650316 (591h12m14.688733351s) [noise=true , address=185.167.204.205:4029 , piece_id=KUOMMYJ2F4NIK6VN4C5M3Y3TMHYEZH7QFNJ6K3CLOAVI5R6VKTHQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:14.817870054 (591h12m14.688953099s) [noise=true , address=185.167.204.205:4019 , piece_id=H6AH654VGX6NHBUKODIWGL7I2RKE3DDMY4UM6VDONP2EP3NPDJBA, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:14.818623698 (591h12m14.689706744s) [noise=true , address=185.167.204.206:4065 , piece_id=LD3J4656Q5QEUCMJ5OKEYATEVRIKSXO7XGARTN25BVM7FDIYOWKA, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:14.818932436 (591h12m14.690015481s) [noise=true , address=185.167.204.206:4061 , piece_id=T2XS4IFPQSVS3T45SPQ5ZDDVT2VMTMXHKL2ZMX4OIOPRR67SCD2Q, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:14.819147046 (591h12m14.690230091s) [noise=true , address=185.167.204.206:4071 , piece_id=QI2TOY5KCDZVWCCLMAJXVUWD2HB7S7IOTEAV2EHAMUNHKOFKTNYA, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:14.819678578 (591h12m14.690761623s) [noise=true , address=185.167.204.206:4059 , piece_id=LMOGTWX2YIRR3EKNROK43O2B27S5L5U6GYB7HBVRBPQPH4WDDWZQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:14.819905028 (591h12m14.690988063s) [noise=true , address=185.167.204.206:4058 , piece_id=ROOCGK64M5RHIWNAVCSZNYTZ6M2ORWB7MJ27XJVDTFVQIKJZHY4A, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:14.820479312 (591h12m14.691562357s) [noise=true , address=185.167.204.205:4021 , piece_id=KP7J5XSY4SKSKZG7XXFZMNKMRP4472PXI7CW2UQBZICN4MAFO5RQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:14.823149296 (591h12m14.694232331s) [noise=true , address=185.167.204.205:4034 , piece_id=OWZHQB5ZDT2QZ2QAHZJVRWQ222MX6CWLKJUOIAAGSMAVLWGJAOBA, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:14.827906996 (591h12m14.698990041s) [noise=true , address=185.167.204.206:4055 , piece_id=VYEDR6XHZBZMTVJ5U5OF67WZOGVMCMQLXXBQUI7DHBCCVQ2RRERQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:14.828003769 (591h12m14.699086814s) [noise=true , address=185.167.204.205:4016 , piece_id=I6MUDLAXUUDJL2C4HFE22C2L56A7D6HSCNWXKWJ7OKO2N4RA4W7Q, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:14.828893203 (591h12m14.699976238s) [noise=true , address=185.167.204.206:4064 , piece_id=H4PP7HPHXRX2LOZ4776AMWAJDN7OC435E2K4BXI7XOZYKTB4YUNA, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:14.829477074 (591h12m14.700560109s) [noise=true , address=185.167.204.205:4017 , piece_id=CZP65KETMGOHFDMV7KENRL4V36UHBPWRDCBBPUM5NE5L2PV4AQCQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:14.829774972 (591h12m14.700858007s) [noise=true , address=185.167.204.206:4043 , piece_id=G7VZYID4TXGNPDN63NQULXOVSX42WMFFNPBX6KQJ5L2WSW4D2YGQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:14.831734493 (591h12m14.702817528s) [noise=true , address=185.167.204.206:4049 , piece_id=BQ7JEGSGPIFJUGZFBYRPDMBNPZ24BG6TEG4VUM4JUBTE4T2RF4SA, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:15.405688561 (591h12m15.276771606s) [noise=true , address=81.171.14.70:4027 , piece_id=6WEORWX5QOMFSJUEADIIBHHIPKNFP4NJPZAJADBUTPUSG7MZJMOQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:15.417110440 (591h12m15.288193485s) [noise=true , address=81.171.14.70:4031 , piece_id=U6Z4SNK32ZSWQ7NDR7ACXGRC4BADDOPR65ILBR64QLGVNDBADFFQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:15.425600446 (591h12m15.296683481s) [noise=true , address=81.171.14.70:4037 , piece_id=6A25LEIAQEJB3DNCTXQT6K7ZSKIGOW4QUGMNKLOG3CT2XXH25PSA, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:15.499056228 (591h12m15.370139263s) [noise=true , address=81.171.14.70:4038 , piece_id=3WRHW3VGXWHOEOV5Q6CLYSF7MUVWS7QHTNHLSTCKAUERPHAH3I3Q, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:15.514998925 (591h12m15.38608197s) [noise=true , address=81.171.14.70:4041 , piece_id=NORNA6WFTGPNOHU7MPTNNB5ONTBS5DJGCN2NWIZAEPYSUSIOSGEQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:15.528579544 (591h12m15.399662589s) [noise=true , address=81.171.14.70:4030 , piece_id=KITOJIT34CHS2QZBXCW2SJPZB4RHEHCMJ6IYQ4Z5KLL24YAWBIUQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:15.528991398 (591h12m15.400074434s) [noise=true , address=81.171.14.70:4047 , piece_id=ZWV5DP4YC3MSYDWEQH4ZK3VR4WZ4DUTI4VL3WVLQKFIEKIW44LOQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:15.534844544 (591h12m15.405927579s) [noise=true , address=81.171.14.70:4035 , piece_id=XCLL5SOAPJSSGF5YYQH6ZKGOHMLS6RI6H26FOW5XIJRY3TWYNEGA, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:15.915614910 (591h12m15.786697945s) [noise=true , address=81.171.14.70:4039 , piece_id=DEJDXFDHOSAMIP2FN5LJBJNBB4QLJSE2AUDW6IMA6IC4VRY7VAGQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:15.967820176 (591h12m15.838903211s) [noise=true , address=81.171.14.70:4046 , piece_id=B2MTUNKMUP5S7XW4SVQ5QI2ZO3VFB5QVAKBBY4BIZIUILM6O7VYQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:16.172116128 (591h12m16.043199163s) [noise=true , address=89.149.201.147:4007 , piece_id=HZLSYGSBP4TJH5V5XF3Q3AETJU625ERNDBD3VXW2HUZTTPAHLBMA, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:16.172137559 (591h12m16.043220604s) [seg_pos={0 0}, upload=518292]: Segment reached optimal threshold of 26 pieces.
Dec 30 06:41:16.172336949 (591h12m16.043419994s) [noise=true , address=89.149.201.147:4008 , piece_id=XYFRNJVIFFZA5S36KFPGZPVCPLRL2OW5QBJD3UR37VPPA2RX3RJA, seg_pos={0 0}, upload=518292]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QN8VP6o7qGw9Dd8EafU9xe7s7Cuf1FyqPMbnRaznrfPxfEUUn): protocol: expected piece hash; context canceled; EOF
Dec 30 06:41:16.172344213 (591h12m16.043427258s) [noise=true , address=89.149.201.147:4008 , piece_id=XYFRNJVIFFZA5S36KFPGZPVCPLRL2OW5QBJD3UR37VPPA2RX3RJA, seg_pos={0 0}, upload=518292]: Long tail cancellation due to optimal threshold
Dec 30 06:41:16.172554002 (591h12m16.043637037s) [noise=true , address=89.149.201.147:4020 , piece_id=N7G5DGEDYVKBTO5S6BYW3IYEOK7HAMK5JQIBMVPMPYRDHXODWEGA, seg_pos={0 0}, upload=518292]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12YeKHJH7CoouLFJYENJhzVLNkA32hJaqjLz3dXhr78eaSGtcic): protocol: expected piece hash; context canceled; EOF
Dec 30 06:41:16.172565233 (591h12m16.043648278s) [noise=true , address=89.149.201.147:4020 , piece_id=N7G5DGEDYVKBTO5S6BYW3IYEOK7HAMK5JQIBMVPMPYRDHXODWEGA, seg_pos={0 0}, upload=518292]: Long tail cancellation due to optimal threshold
Dec 30 06:41:16.172587506 (591h12m16.043670541s) [seg_pos={0 0}, upload=518292]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil>
Dec 30 06:41:16.172717945 (591h12m16.04380098s) [upload=518292]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 06:41:16.172724567 (591h12m16.043807602s) [upload=518292]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 06:41:16.404092237 (591h12m16.275175292s) [upload=518292]: Done uploading segments...
Dec 30 06:41:16.404119940 (591h12m16.275202975s) [upload=518292]: Upload finished. err: <nil>
Duration: 6.511150797s
core
uplink
eu-select-1
eu1-placement-15
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
9 out of 60 is failed (threshold=6) [....X...........X..............XX...X......X..X..X.....X....] duration:
3.658793312s trace:
https://jaeger.dev.storj.io/trace/7d0f3c444f9c906c
FAIL (2025-12-30 06:45:41 UTC)
orbiter warning: context deadline exceeded
Duration: 3.658819463s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [................X...........................................] duration:
16.159117287s trace:
https://jaeger.dev.storj.io/trace/deb96436003dee
FAIL (2025-12-30 06:45:38 UTC)
orbiter warning: read tcp 188.166.225.170:52458->45.38.24.2:443: read: connection reset by peer
Duration: 16.159146553s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.................X..........................................] duration:
19.451767498s trace:
https://jaeger.dev.storj.io/trace/deb96436003dee
FAIL (2025-12-30 06:45:16 UTC)
orbiter warning: read tcp 188.166.225.170:52458->45.38.24.2:443: read: connection reset by peer
Duration: 19.451796694s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
4 out of 60 is failed (threshold=3) [......X...X......X............X.............................] duration:
29.611155181s trace:
https://jaeger.dev.storj.io/trace/45b769fc71e13448
FAIL (2025-12-30 06:45:14 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: context deadline exceeded
======LOGS======
.577697683 (590h53m11.448780728s) [noise=true , address=193.149.164.143:4704 , piece_id=4T2IJIA6WHI5LMIUP5BLJLCD3GRY3A6PXR6G5OWJWUSYZT5ELGHQ, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.578173528 (590h53m11.449256573s) [noise=true , address=64.49.13.27:4965 , piece_id=CO4O5F2IBZAAKBZBV3HWNNOXZOGLNNDPALWMQZ3DEM3MREASVFEA, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.578542992 (590h53m11.449626027s) [noise=true , address=77.90.31.15:4729 , piece_id=NVMXFEILHGT622IIPYYT4FFHC2WZLHMCZM4VT5WUPHRAPH3DX6TA, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.578846911 (590h53m11.449929946s) [noise=true , address=145.79.227.43:4177 , piece_id=YM37ZI3WNZ7SXGTFC224TOVVTHYNFIPRQB2TFLEZ735RNZE553FQ, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.579458044 (590h53m11.450541089s) [noise=true , address=45.91.116.11:4150 , piece_id=E66IDHTJDLYLYJIOWMOIKOOWLXCUNCMBIQ7SQBAD3VTJF6VX5JLQ, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.579891209 (590h53m11.450974245s) [noise=true , address=145.79.227.29:4945 , piece_id=SHL73IDEI54GRCWYNWHRUSJZZ4UUXREAX54DFJLTG5QPZCF5QKIQ, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.579915535 (590h53m11.45099858s) [noise=true , address=145.79.227.30:4772 , piece_id=PVMD5DWRRPVARTSZ6O7HN6MUVTZUYWLYSNQR5GDMNKUJZGXMYPFQ, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.580325466 (590h53m11.451408511s) [noise=true , address=77.90.31.12:4603 , piece_id=2I4L7F2WEGK5OSLODJSJ4672URBA7DUGXWZKTYB6ZNA2TQIGVXWA, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.580379599 (590h53m11.451462644s) [noise=true , address=77.90.31.14:4671 , piece_id=263TRISJ5ADFTX43ULSSGBAF2KMOXAPISJ56LLE53D74XOEMQ5EA, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.580692044 (590h53m11.451775089s) [noise=true , address=145.79.227.3:4175 , piece_id=UCV5WQNVWEAMEDBWDNXEVSQEGXPJE7KD6YSWC5A4WXYUKUCK4NLA, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.580857859 (590h53m11.451940894s) [noise=true , address=77.90.31.19:4860 , piece_id=U2MNFOAD7FHCDBE4FUYOROIP3P33ROJ7K75SF5Z747EFKFTF7RGA, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.581089210 (590h53m11.452172255s) [noise=true , address=145.79.227.44:4203 , piece_id=WHCKQ2FRLYAZE7Q6HYZNTS4EISVHZNAPDDKH34WN33ALK4MVK2EA, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.581351320 (590h53m11.452434355s) [noise=true , address=64.49.13.154:4928 , piece_id=SEVBZBMNGURP5TRUEBA4PHRSA6MNANVJUKINALJUWPQWBGKVYNTA, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.581457091 (590h53m11.452540126s) [noise=true , address=145.79.227.34:4909 , piece_id=3QSWTNR5LGUUAY3N5HJKSPL3TRQ6WO5F5OD4VTXKWHWUXUF7PVVQ, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.581863144 (590h53m11.452946179s) [noise=true , address=145.79.227.33:4881 , piece_id=EX66JF7U7A4ZSRWJMP4MCLQRPW2ZZFZFIUNV72XZXU7E5O3KWX3A, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.584010443 (590h53m11.455093478s) [noise=true , address=145.79.227.40:4070 , piece_id=YOAGDXNN4TTCK35LLQDSP24BOWLZXGNSQU7WMKEEVVMPXDVI74JA, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.585122791 (590h53m11.456205826s) [noise=true , address=45.91.116.14:4217 , piece_id=L4MAHTGKTNWBSN3YTB74E3DBJWVMA3DPU57MQK5FNK35DXRZ6AIA, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:12.983305402 (590h53m12.854388447s) [upload=517998]: Next returned nil segment
Dec 30 06:22:12.983321313 (590h53m12.854404348s) [upload=517998]: Waiting for error group managing segments...
Dec 30 06:22:13.379703053 (590h53m13.250786088s) [noise=true , address=64.49.13.27:4965 , piece_id=CO4O5F2IBZAAKBZBV3HWNNOXZOGLNNDPALWMQZ3DEM3MREASVFEA, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:13.418055622 (590h53m13.289138657s) [noise=true , address=145.79.227.3:4175 , piece_id=UCV5WQNVWEAMEDBWDNXEVSQEGXPJE7KD6YSWC5A4WXYUKUCK4NLA, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:13.509034160 (590h53m13.380117205s) [noise=true , address=64.49.13.154:4928 , piece_id=SEVBZBMNGURP5TRUEBA4PHRSA6MNANVJUKINALJUWPQWBGKVYNTA, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:13.642886154 (590h53m13.513969199s) [noise=true , address=77.90.31.14:4671 , piece_id=263TRISJ5ADFTX43ULSSGBAF2KMOXAPISJ56LLE53D74XOEMQ5EA, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:13.788145520 (590h53m13.659228555s) [noise=true , address=77.90.31.12:4603 , piece_id=2I4L7F2WEGK5OSLODJSJ4672URBA7DUGXWZKTYB6ZNA2TQIGVXWA, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:13.858878757 (590h53m13.729961792s) [noise=true , address=145.79.227.49:4369 , piece_id=XUEXGNJJIJ4DUBDNUV7CKF6J5JYHSLZAEPVGEDGBUBY42BIP4XEQ, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:13.879550569 (590h53m13.750633614s) [noise=true , address=145.79.227.34:4909 , piece_id=3QSWTNR5LGUUAY3N5HJKSPL3TRQ6WO5F5OD4VTXKWHWUXUF7PVVQ, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:13.940897723 (590h53m13.811980769s) [noise=true , address=77.90.31.19:4860 , piece_id=U2MNFOAD7FHCDBE4FUYOROIP3P33ROJ7K75SF5Z747EFKFTF7RGA, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:13.989791936 (590h53m13.860874981s) [noise=true , address=45.91.116.11:4150 , piece_id=E66IDHTJDLYLYJIOWMOIKOOWLXCUNCMBIQ7SQBAD3VTJF6VX5JLQ, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:14.289516718 (590h53m14.160599753s) [noise=true , address=145.79.227.44:4203 , piece_id=WHCKQ2FRLYAZE7Q6HYZNTS4EISVHZNAPDDKH34WN33ALK4MVK2EA, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:14.338651537 (590h53m14.209734582s) [noise=true , address=77.90.31.15:4729 , piece_id=NVMXFEILHGT622IIPYYT4FFHC2WZLHMCZM4VT5WUPHRAPH3DX6TA, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:14.344145168 (590h53m14.215228213s) [noise=true , address=145.79.227.30:4772 , piece_id=PVMD5DWRRPVARTSZ6O7HN6MUVTZUYWLYSNQR5GDMNKUJZGXMYPFQ, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:14.667232847 (590h53m14.538315892s) [noise=true , address=193.149.164.143:4704 , piece_id=4T2IJIA6WHI5LMIUP5BLJLCD3GRY3A6PXR6G5OWJWUSYZT5ELGHQ, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:14.671388990 (590h53m14.542472025s) [noise=true , address=145.79.227.29:4945 , piece_id=SHL73IDEI54GRCWYNWHRUSJZZ4UUXREAX54DFJLTG5QPZCF5QKIQ, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:14.671415190 (590h53m14.542498226s) [seg_pos={0 0}, upload=517998]: Segment reached optimal threshold of 14 pieces.
Dec 30 06:22:14.671803600 (590h53m14.542886655s) [noise=true , address=145.79.227.40:4070 , piece_id=YOAGDXNN4TTCK35LLQDSP24BOWLZXGNSQU7WMKEEVVMPXDVI74JA, seg_pos={0 0}, upload=517998]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12hU1LwL2gX7rVnrNgzaBCYB37ziqf7s7aUgtF2Uc3antReQ6me): protocol: expected piece hash; context canceled; EOF
Dec 30 06:22:14.671814470 (590h53m14.542897525s) [noise=true , address=145.79.227.40:4070 , piece_id=YOAGDXNN4TTCK35LLQDSP24BOWLZXGNSQU7WMKEEVVMPXDVI74JA, seg_pos={0 0}, upload=517998]: Long tail cancellation due to optimal threshold
Dec 30 06:22:14.672014932 (590h53m14.543097977s) [noise=true , address=145.79.227.43:4177 , piece_id=YM37ZI3WNZ7SXGTFC224TOVVTHYNFIPRQB2TFLEZ735RNZE553FQ, seg_pos={0 0}, upload=517998]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bhN3cR22QuAYihnnB3x3R97UZRWJ8sFPP8fjQcXZ7qhNZphZG): protocol: expected piece hash; context canceled; EOF
Dec 30 06:22:14.672022637 (590h53m14.543105672s) [noise=true , address=145.79.227.43:4177 , piece_id=YM37ZI3WNZ7SXGTFC224TOVVTHYNFIPRQB2TFLEZ735RNZE553FQ, seg_pos={0 0}, upload=517998]: Long tail cancellation due to optimal threshold
Dec 30 06:22:14.672088933 (590h53m14.543171988s) [noise=true , address=145.79.227.33:4881 , piece_id=EX66JF7U7A4ZSRWJMP4MCLQRPW2ZZFZFIUNV72XZXU7E5O3KWX3A, seg_pos={0 0}, upload=517998]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mMBjg1b4MqTZPE5NpNNMYDzmsHqWYotE5jBNmZLyA5FyYLZRp): protocol: expected piece hash; context canceled; EOF
Dec 30 06:22:14.672127677 (590h53m14.543210742s) [noise=true , address=145.79.227.33:4881 , piece_id=EX66JF7U7A4ZSRWJMP4MCLQRPW2ZZFZFIUNV72XZXU7E5O3KWX3A, seg_pos={0 0}, upload=517998]: Long tail cancellation due to optimal threshold
Dec 30 06:22:14.672312739 (590h53m14.543395774s) [noise=true , address=45.91.116.14:4217 , piece_id=L4MAHTGKTNWBSN3YTB74E3DBJWVMA3DPU57MQK5FNK35DXRZ6AIA, seg_pos={0 0}, upload=517998]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xSXWFiq27Dd2vBDD9vVnhDHVG59UmDKn3hVAhf3oyo5goVNaE): protocol: expected piece hash; context canceled; EOF
Dec 30 06:22:14.672319953 (590h53m14.543402988s) [noise=true , address=45.91.116.14:4217 , piece_id=L4MAHTGKTNWBSN3YTB74E3DBJWVMA3DPU57MQK5FNK35DXRZ6AIA, seg_pos={0 0}, upload=517998]: Long tail cancellation due to optimal threshold
Dec 30 06:22:14.672337847 (590h53m14.543420882s) [seg_pos={0 0}, upload=517998]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 30 06:22:14.672452947 (590h53m14.543535982s) [upload=517998]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 06:22:14.672458507 (590h53m14.543541542s) [upload=517998]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 06:22:14.981437069 (590h53m14.852520114s) [upload=517998]: Done uploading segments...
Dec 30 06:22:14.981456546 (590h53m14.852539581s) [upload=517998]: Upload finished. err: <nil>
Duration: 29.611168807s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)antiblip:
1 out of 60 is failed (threshold=3) [.......................................................X....] duration:
6.555119256s trace:
https://jaeger.dev.storj.io/trace/4c2f4efaff0e0ea0
FAIL (2025-12-30 06:44:59 UTC)
orbiter warning:
operation: DeleteObject
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
load=518292]: Uploading piece...
Dec 30 06:41:13.598100152 (591h12m13.469183187s) [noise=true , address=81.171.14.70:4041 , piece_id=NORNA6WFTGPNOHU7MPTNNB5ONTBS5DJGCN2NWIZAEPYSUSIOSGEQ, seg_pos={0 0}, upload=518292]: Uploading piece...
Dec 30 06:41:13.598587971 (591h12m13.469671016s) [noise=true , address=89.149.201.147:4007 , piece_id=HZLSYGSBP4TJH5V5XF3Q3AETJU625ERNDBD3VXW2HUZTTPAHLBMA, seg_pos={0 0}, upload=518292]: Uploading piece...
Dec 30 06:41:13.599093143 (591h12m13.470176178s) [noise=true , address=89.149.201.147:4008 , piece_id=XYFRNJVIFFZA5S36KFPGZPVCPLRL2OW5QBJD3UR37VPPA2RX3RJA, seg_pos={0 0}, upload=518292]: Uploading piece...
Dec 30 06:41:13.599699999 (591h12m13.470783034s) [noise=true , address=81.171.14.70:4039 , piece_id=DEJDXFDHOSAMIP2FN5LJBJNBB4QLJSE2AUDW6IMA6IC4VRY7VAGQ, seg_pos={0 0}, upload=518292]: Uploading piece...
Dec 30 06:41:13.600136169 (591h12m13.471219204s) [noise=true , address=185.167.204.205:4034 , piece_id=OWZHQB5ZDT2QZ2QAHZJVRWQ222MX6CWLKJUOIAAGSMAVLWGJAOBA, seg_pos={0 0}, upload=518292]: Uploading piece...
Dec 30 06:41:13.600543184 (591h12m13.471626229s) [noise=true , address=185.167.204.206:4058 , piece_id=ROOCGK64M5RHIWNAVCSZNYTZ6M2ORWB7MJ27XJVDTFVQIKJZHY4A, seg_pos={0 0}, upload=518292]: Uploading piece...
Dec 30 06:41:13.601048286 (591h12m13.472131332s) [noise=true , address=81.171.14.70:4046 , piece_id=B2MTUNKMUP5S7XW4SVQ5QI2ZO3VFB5QVAKBBY4BIZIUILM6O7VYQ, seg_pos={0 0}, upload=518292]: Uploading piece...
Dec 30 06:41:13.601698784 (591h12m13.472781819s) [noise=true , address=185.167.204.206:4064 , piece_id=H4PP7HPHXRX2LOZ4776AMWAJDN7OC435E2K4BXI7XOZYKTB4YUNA, seg_pos={0 0}, upload=518292]: Uploading piece...
Dec 30 06:41:13.602168259 (591h12m13.473251304s) [noise=true , address=185.167.204.206:4065 , piece_id=LD3J4656Q5QEUCMJ5OKEYATEVRIKSXO7XGARTN25BVM7FDIYOWKA, seg_pos={0 0}, upload=518292]: Uploading piece...
Dec 30 06:41:13.602782708 (591h12m13.473865753s) [noise=true , address=81.171.14.70:4027 , piece_id=6WEORWX5QOMFSJUEADIIBHHIPKNFP4NJPZAJADBUTPUSG7MZJMOQ, seg_pos={0 0}, upload=518292]: Uploading piece...
Dec 30 06:41:14.423395002 (591h12m14.294478047s) [upload=518292]: Next returned nil segment
Dec 30 06:41:14.423414068 (591h12m14.294497103s) [upload=518292]: Waiting for error group managing segments...
Dec 30 06:41:14.817650316 (591h12m14.688733351s) [noise=true , address=185.167.204.205:4029 , piece_id=KUOMMYJ2F4NIK6VN4C5M3Y3TMHYEZH7QFNJ6K3CLOAVI5R6VKTHQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:14.817870054 (591h12m14.688953099s) [noise=true , address=185.167.204.205:4019 , piece_id=H6AH654VGX6NHBUKODIWGL7I2RKE3DDMY4UM6VDONP2EP3NPDJBA, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:14.818623698 (591h12m14.689706744s) [noise=true , address=185.167.204.206:4065 , piece_id=LD3J4656Q5QEUCMJ5OKEYATEVRIKSXO7XGARTN25BVM7FDIYOWKA, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:14.818932436 (591h12m14.690015481s) [noise=true , address=185.167.204.206:4061 , piece_id=T2XS4IFPQSVS3T45SPQ5ZDDVT2VMTMXHKL2ZMX4OIOPRR67SCD2Q, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:14.819147046 (591h12m14.690230091s) [noise=true , address=185.167.204.206:4071 , piece_id=QI2TOY5KCDZVWCCLMAJXVUWD2HB7S7IOTEAV2EHAMUNHKOFKTNYA, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:14.819678578 (591h12m14.690761623s) [noise=true , address=185.167.204.206:4059 , piece_id=LMOGTWX2YIRR3EKNROK43O2B27S5L5U6GYB7HBVRBPQPH4WDDWZQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:14.819905028 (591h12m14.690988063s) [noise=true , address=185.167.204.206:4058 , piece_id=ROOCGK64M5RHIWNAVCSZNYTZ6M2ORWB7MJ27XJVDTFVQIKJZHY4A, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:14.820479312 (591h12m14.691562357s) [noise=true , address=185.167.204.205:4021 , piece_id=KP7J5XSY4SKSKZG7XXFZMNKMRP4472PXI7CW2UQBZICN4MAFO5RQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:14.823149296 (591h12m14.694232331s) [noise=true , address=185.167.204.205:4034 , piece_id=OWZHQB5ZDT2QZ2QAHZJVRWQ222MX6CWLKJUOIAAGSMAVLWGJAOBA, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:14.827906996 (591h12m14.698990041s) [noise=true , address=185.167.204.206:4055 , piece_id=VYEDR6XHZBZMTVJ5U5OF67WZOGVMCMQLXXBQUI7DHBCCVQ2RRERQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:14.828003769 (591h12m14.699086814s) [noise=true , address=185.167.204.205:4016 , piece_id=I6MUDLAXUUDJL2C4HFE22C2L56A7D6HSCNWXKWJ7OKO2N4RA4W7Q, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:14.828893203 (591h12m14.699976238s) [noise=true , address=185.167.204.206:4064 , piece_id=H4PP7HPHXRX2LOZ4776AMWAJDN7OC435E2K4BXI7XOZYKTB4YUNA, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:14.829477074 (591h12m14.700560109s) [noise=true , address=185.167.204.205:4017 , piece_id=CZP65KETMGOHFDMV7KENRL4V36UHBPWRDCBBPUM5NE5L2PV4AQCQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:14.829774972 (591h12m14.700858007s) [noise=true , address=185.167.204.206:4043 , piece_id=G7VZYID4TXGNPDN63NQULXOVSX42WMFFNPBX6KQJ5L2WSW4D2YGQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:14.831734493 (591h12m14.702817528s) [noise=true , address=185.167.204.206:4049 , piece_id=BQ7JEGSGPIFJUGZFBYRPDMBNPZ24BG6TEG4VUM4JUBTE4T2RF4SA, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:15.405688561 (591h12m15.276771606s) [noise=true , address=81.171.14.70:4027 , piece_id=6WEORWX5QOMFSJUEADIIBHHIPKNFP4NJPZAJADBUTPUSG7MZJMOQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:15.417110440 (591h12m15.288193485s) [noise=true , address=81.171.14.70:4031 , piece_id=U6Z4SNK32ZSWQ7NDR7ACXGRC4BADDOPR65ILBR64QLGVNDBADFFQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:15.425600446 (591h12m15.296683481s) [noise=true , address=81.171.14.70:4037 , piece_id=6A25LEIAQEJB3DNCTXQT6K7ZSKIGOW4QUGMNKLOG3CT2XXH25PSA, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:15.499056228 (591h12m15.370139263s) [noise=true , address=81.171.14.70:4038 , piece_id=3WRHW3VGXWHOEOV5Q6CLYSF7MUVWS7QHTNHLSTCKAUERPHAH3I3Q, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:15.514998925 (591h12m15.38608197s) [noise=true , address=81.171.14.70:4041 , piece_id=NORNA6WFTGPNOHU7MPTNNB5ONTBS5DJGCN2NWIZAEPYSUSIOSGEQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:15.528579544 (591h12m15.399662589s) [noise=true , address=81.171.14.70:4030 , piece_id=KITOJIT34CHS2QZBXCW2SJPZB4RHEHCMJ6IYQ4Z5KLL24YAWBIUQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:15.528991398 (591h12m15.400074434s) [noise=true , address=81.171.14.70:4047 , piece_id=ZWV5DP4YC3MSYDWEQH4ZK3VR4WZ4DUTI4VL3WVLQKFIEKIW44LOQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:15.534844544 (591h12m15.405927579s) [noise=true , address=81.171.14.70:4035 , piece_id=XCLL5SOAPJSSGF5YYQH6ZKGOHMLS6RI6H26FOW5XIJRY3TWYNEGA, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:15.915614910 (591h12m15.786697945s) [noise=true , address=81.171.14.70:4039 , piece_id=DEJDXFDHOSAMIP2FN5LJBJNBB4QLJSE2AUDW6IMA6IC4VRY7VAGQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:15.967820176 (591h12m15.838903211s) [noise=true , address=81.171.14.70:4046 , piece_id=B2MTUNKMUP5S7XW4SVQ5QI2ZO3VFB5QVAKBBY4BIZIUILM6O7VYQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:16.172116128 (591h12m16.043199163s) [noise=true , address=89.149.201.147:4007 , piece_id=HZLSYGSBP4TJH5V5XF3Q3AETJU625ERNDBD3VXW2HUZTTPAHLBMA, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:16.172137559 (591h12m16.043220604s) [seg_pos={0 0}, upload=518292]: Segment reached optimal threshold of 26 pieces.
Dec 30 06:41:16.172336949 (591h12m16.043419994s) [noise=true , address=89.149.201.147:4008 , piece_id=XYFRNJVIFFZA5S36KFPGZPVCPLRL2OW5QBJD3UR37VPPA2RX3RJA, seg_pos={0 0}, upload=518292]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QN8VP6o7qGw9Dd8EafU9xe7s7Cuf1FyqPMbnRaznrfPxfEUUn): protocol: expected piece hash; context canceled; EOF
Dec 30 06:41:16.172344213 (591h12m16.043427258s) [noise=true , address=89.149.201.147:4008 , piece_id=XYFRNJVIFFZA5S36KFPGZPVCPLRL2OW5QBJD3UR37VPPA2RX3RJA, seg_pos={0 0}, upload=518292]: Long tail cancellation due to optimal threshold
Dec 30 06:41:16.172554002 (591h12m16.043637037s) [noise=true , address=89.149.201.147:4020 , piece_id=N7G5DGEDYVKBTO5S6BYW3IYEOK7HAMK5JQIBMVPMPYRDHXODWEGA, seg_pos={0 0}, upload=518292]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12YeKHJH7CoouLFJYENJhzVLNkA32hJaqjLz3dXhr78eaSGtcic): protocol: expected piece hash; context canceled; EOF
Dec 30 06:41:16.172565233 (591h12m16.043648278s) [noise=true , address=89.149.201.147:4020 , piece_id=N7G5DGEDYVKBTO5S6BYW3IYEOK7HAMK5JQIBMVPMPYRDHXODWEGA, seg_pos={0 0}, upload=518292]: Long tail cancellation due to optimal threshold
Dec 30 06:41:16.172587506 (591h12m16.043670541s) [seg_pos={0 0}, upload=518292]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil>
Dec 30 06:41:16.172717945 (591h12m16.04380098s) [upload=518292]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 06:41:16.172724567 (591h12m16.043807602s) [upload=518292]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 06:41:16.404092237 (591h12m16.275175292s) [upload=518292]: Done uploading segments...
Dec 30 06:41:16.404119940 (591h12m16.275202975s) [upload=518292]: Upload finished. err: <nil>
Duration: 6.555160545s
core
uplink
eu-select-1
eu1-placement-15
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
14 out of 60 is failed (threshold=3) [....X..X......X....X...X..X.X.X.....X.......X.X.....X....XX.] duration:
5.914171391s trace:
https://jaeger.dev.storj.io/trace/78056e8d4b3848bb
FAIL (2025-12-30 06:44:54 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
Dec 30 06:43:43.438454167 (591h14m43.309537212s) [upload=518325]: Starting upload
Dec 30 06:43:43.438541793 (591h14m43.309624849s) [upload=518325]: Done starting upload
Dec 30 06:43:43.438670769 (591h14m43.309753824s) [upload=518325]: Uploading segments...
Dec 30 06:43:43.438849339 (591h14m43.309932394s) [upload=518325]: Got next segment. Inline: false
Dec 30 06:43:43.438883805 (591h14m43.30996685s) [upload=518325]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 30 06:43:43.438889526 (591h14m43.309972561s) [upload=518325]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 30 06:43:43.682039201 (591h14m43.553122246s) [seg_pos={0 0}, upload=518325]: Begin upload segment...
Dec 30 06:43:43.682106038 (591h14m43.553189083s) [seg_pos={0 0}, upload=518325]: Done begin upload segment.
Dec 30 06:43:43.682152857 (591h14m43.553235902s) [noise=true , address=145.79.227.11:4732 , piece_id=WOMYTSZBK7KZRKDYYZA3ORJ5M6RCYJUHU2O7GXYY3CSQVFYE3S4A, seg_pos={0 0}, upload=518325]: Uploading piece...
Dec 30 06:43:43.682258869 (591h14m43.553341914s) [noise=true , address=77.90.31.17:4770 , piece_id=EZYUKNUOTNWRXEROZAVMPDNQ2EAYUBO3KP7X2GWSQXU6ZRQ5EJQQ, seg_pos={0 0}, upload=518325]: Uploading piece...
Dec 30 06:43:43.683367860 (591h14m43.554450906s) [noise=true , address=145.79.227.14:4838 , piece_id=LNTWJQBCQVT7WXSCH5EIII7DHI4SVOYIVKK4F7IXIKM4KMED2YMQ, seg_pos={0 0}, upload=518325]: Uploading piece...
Dec 30 06:43:43.683377639 (591h14m43.554460684s) [noise=true , address=145.79.227.36:4518 , piece_id=KWGCY63F2DZC5MNJIY5T73RQHOGDUNIOHOMFBXJXWIQXFQA73PYA, seg_pos={0 0}, upload=518325]: Uploading piece...
Dec 30 06:43:43.683832937 (591h14m43.554915972s) [noise=true , address=45.91.116.11:4120 , piece_id=VTJS4ZRPH6KEIEAYEQ4ODRTTGTX6BA2FWMZRTNT75OF7SH2DZUDA, seg_pos={0 0}, upload=518325]: Uploading piece...
Dec 30 06:43:43.684783196 (591h14m43.555866241s) [noise=true , address=145.79.227.18:4007 , piece_id=OWWYNA55RHUTUGUQT4Y2ZBI3VQAQQVNJTFKG6ALMRXXDLW3DW4AQ, seg_pos={0 0}, upload=518325]: Uploading piece...
Dec 30 06:43:43.685038492 (591h14m43.556121527s) [noise=true , address=145.79.227.3:4151 , piece_id=EJ6XIYN4PQ5KGO6EYP4YMQHXEZKZ2NHASAJTNVQ3VF7S5PNHCYKA, seg_pos={0 0}, upload=518325]: Uploading piece...
Dec 30 06:43:43.686167853 (591h14m43.557250908s) [noise=true , address=45.91.116.9:4263 , piece_id=AS5TS3VJSLQMF7ZHLF7B2VBTDVIB23LHUU63EGI2ENDXGEYPDV5A, seg_pos={0 0}, upload=518325]: Uploading piece...
Dec 30 06:43:43.686258695 (591h14m43.55734174s) [noise=true , address=145.79.227.29:4943 , piece_id=TDSZ3VSCCNAU73ZL2CS5XPWOCA6TRTZZKY7VDS2WSK3E2PFFTNMA, seg_pos={0 0}, upload=518325]: Uploading piece...
Dec 30 06:43:43.687362998 (591h14m43.558446043s) [noise=true , address=45.91.116.10:4083 , piece_id=AFTCCUF3L7226XLTU4GEFEKYGYQAQPJKHIPKX5N7MT2UCU45L2BQ, seg_pos={0 0}, upload=518325]: Uploading piece...
Dec 30 06:43:43.687636299 (591h14m43.558719334s) [noise=true , address=145.79.227.50:4374 , piece_id=W3NSQQDQ72JDIVU3LX7BQSWTCTEM3UAZLH2GU2AM7I7DWIRHFKZA, seg_pos={0 0}, upload=518325]: Uploading piece...
Dec 30 06:43:43.688312987 (591h14m43.559396022s) [noise=true , address=145.79.227.26:4654 , piece_id=J7RRD6V3GTKDEZEPPOUNPBAMD3Q4QHIG7T6Z7FE72EJAJKSDTGBQ, seg_pos={0 0}, upload=518325]: Uploading piece...
Dec 30 06:43:43.688619340 (591h14m43.559702385s) [noise=true , address=77.90.31.15:4697 , piece_id=ZQJMG5N6Z5DYDTEMF5EL5YQGQVGKPEWMG6RZN6EXMEJYXJUQT3WA, seg_pos={0 0}, upload=518325]: Uploading piece...
Dec 30 06:43:43.688729590 (591h14m43.559812635s) [noise=true , address=145.79.227.6:4257 , piece_id=IMXIK3WJDAMAF7QXSYTWHGLDHUGQIBXQ3WDIZX45L4ODC35H7ICQ, seg_pos={0 0}, upload=518325]: Uploading piece...
Dec 30 06:43:43.689649502 (591h14m43.560732537s) [noise=true , address=77.90.31.10:4517 , piece_id=JXYAH2NNMGP5LSLNUGVUCG3X5NDPBW3MQ6ISE6HV4XPY7Y2CKSNA, seg_pos={0 0}, upload=518325]: Uploading piece...
Dec 30 06:43:43.690303357 (591h14m43.561386402s) [noise=true , address=145.79.227.51:4410 , piece_id=HDGHD2ORSRHHSIJG23HYPYM7GATOSAOD24MYETBMKCICOSTFCJMA, seg_pos={0 0}, upload=518325]: Uploading piece...
Dec 30 06:43:45.296790318 (591h14m45.167873363s) [upload=518325]: Next returned nil segment
Dec 30 06:43:45.296952557 (591h14m45.168035602s) [upload=518325]: Waiting for error group managing segments...
Dec 30 06:43:45.672927161 (591h14m45.544010236s) [noise=true , address=145.79.227.50:4374 , piece_id=W3NSQQDQ72JDIVU3LX7BQSWTCTEM3UAZLH2GU2AM7I7DWIRHFKZA, seg_pos={0 0}, upload=518325]: Done uploading piece. err: <nil>
Dec 30 06:43:45.679981644 (591h14m45.551064679s) [noise=true , address=145.79.227.14:4838 , piece_id=LNTWJQBCQVT7WXSCH5EIII7DHI4SVOYIVKK4F7IXIKM4KMED2YMQ, seg_pos={0 0}, upload=518325]: Done uploading piece. err: <nil>
Dec 30 06:43:45.681994216 (591h14m45.553077251s) [noise=true , address=145.79.227.11:4732 , piece_id=WOMYTSZBK7KZRKDYYZA3ORJ5M6RCYJUHU2O7GXYY3CSQVFYE3S4A, seg_pos={0 0}, upload=518325]: Done uploading piece. err: <nil>
Dec 30 06:43:45.682769572 (591h14m45.553852607s) [noise=true , address=145.79.227.36:4518 , piece_id=KWGCY63F2DZC5MNJIY5T73RQHOGDUNIOHOMFBXJXWIQXFQA73PYA, seg_pos={0 0}, upload=518325]: Done uploading piece. err: <nil>
Dec 30 06:43:45.685368261 (591h14m45.556451296s) [noise=true , address=145.79.227.51:4410 , piece_id=HDGHD2ORSRHHSIJG23HYPYM7GATOSAOD24MYETBMKCICOSTFCJMA, seg_pos={0 0}, upload=518325]: Done uploading piece. err: <nil>
Dec 30 06:43:45.696508394 (591h14m45.567591429s) [noise=true , address=145.79.227.26:4654 , piece_id=J7RRD6V3GTKDEZEPPOUNPBAMD3Q4QHIG7T6Z7FE72EJAJKSDTGBQ, seg_pos={0 0}, upload=518325]: Done uploading piece. err: <nil>
Dec 30 06:43:45.698096838 (591h14m45.569179883s) [noise=true , address=145.79.227.6:4257 , piece_id=IMXIK3WJDAMAF7QXSYTWHGLDHUGQIBXQ3WDIZX45L4ODC35H7ICQ, seg_pos={0 0}, upload=518325]: Done uploading piece. err: <nil>
Dec 30 06:43:45.699199428 (591h14m45.570282463s) [noise=true , address=145.79.227.18:4007 , piece_id=OWWYNA55RHUTUGUQT4Y2ZBI3VQAQQVNJTFKG6ALMRXXDLW3DW4AQ, seg_pos={0 0}, upload=518325]: Done uploading piece. err: <nil>
Dec 30 06:43:45.758676632 (591h14m45.629759677s) [noise=true , address=145.79.227.3:4151 , piece_id=EJ6XIYN4PQ5KGO6EYP4YMQHXEZKZ2NHASAJTNVQ3VF7S5PNHCYKA, seg_pos={0 0}, upload=518325]: Done uploading piece. err: <nil>
Dec 30 06:43:45.903122208 (591h14m45.774205253s) [noise=true , address=77.90.31.10:4517 , piece_id=JXYAH2NNMGP5LSLNUGVUCG3X5NDPBW3MQ6ISE6HV4XPY7Y2CKSNA, seg_pos={0 0}, upload=518325]: Done uploading piece. err: <nil>
Dec 30 06:43:45.976028733 (591h14m45.847111778s) [noise=true , address=145.79.227.29:4943 , piece_id=TDSZ3VSCCNAU73ZL2CS5XPWOCA6TRTZZKY7VDS2WSK3E2PFFTNMA, seg_pos={0 0}, upload=518325]: Done uploading piece. err: <nil>
Dec 30 06:43:45.982909666 (591h14m45.853992711s) [noise=true , address=77.90.31.17:4770 , piece_id=EZYUKNUOTNWRXEROZAVMPDNQ2EAYUBO3KP7X2GWSQXU6ZRQ5EJQQ, seg_pos={0 0}, upload=518325]: Done uploading piece. err: <nil>
Dec 30 06:43:46.039347542 (591h14m45.910430587s) [noise=true , address=45.91.116.10:4083 , piece_id=AFTCCUF3L7226XLTU4GEFEKYGYQAQPJKHIPKX5N7MT2UCU45L2BQ, seg_pos={0 0}, upload=518325]: Done uploading piece. err: <nil>
Dec 30 06:43:46.039372420 (591h14m45.910455475s) [seg_pos={0 0}, upload=518325]: Segment reached optimal threshold of 13 pieces.
Dec 30 06:43:46.039598750 (591h14m45.910681805s) [noise=true , address=77.90.31.15:4697 , piece_id=ZQJMG5N6Z5DYDTEMF5EL5YQGQVGKPEWMG6RZN6EXMEJYXJUQT3WA, seg_pos={0 0}, upload=518325]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5): protocol: expected piece hash; context canceled; EOF
Dec 30 06:43:46.039606464 (591h14m45.91068951s) [noise=true , address=77.90.31.15:4697 , piece_id=ZQJMG5N6Z5DYDTEMF5EL5YQGQVGKPEWMG6RZN6EXMEJYXJUQT3WA, seg_pos={0 0}, upload=518325]: Long tail cancellation due to optimal threshold
Dec 30 06:43:46.039875217 (591h14m45.910958282s) [noise=true , address=45.91.116.9:4263 , piece_id=AS5TS3VJSLQMF7ZHLF7B2VBTDVIB23LHUU63EGI2ENDXGEYPDV5A, seg_pos={0 0}, upload=518325]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF): protocol: expected piece hash; context canceled; EOF
Dec 30 06:43:46.039883282 (591h14m45.910966327s) [noise=true , address=45.91.116.9:4263 , piece_id=AS5TS3VJSLQMF7ZHLF7B2VBTDVIB23LHUU63EGI2ENDXGEYPDV5A, seg_pos={0 0}, upload=518325]: Long tail cancellation due to optimal threshold
Dec 30 06:43:46.039942615 (591h14m45.91102568s) [noise=true , address=45.91.116.11:4120 , piece_id=VTJS4ZRPH6KEIEAYEQ4ODRTTGTX6BA2FWMZRTNT75OF7SH2DZUDA, seg_pos={0 0}, upload=518325]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8): protocol: expected piece hash; context canceled; EOF
Dec 30 06:43:46.039962774 (591h14m45.911045829s) [noise=true , address=45.91.116.11:4120 , piece_id=VTJS4ZRPH6KEIEAYEQ4ODRTTGTX6BA2FWMZRTNT75OF7SH2DZUDA, seg_pos={0 0}, upload=518325]: Long tail cancellation due to optimal threshold
Dec 30 06:43:46.039995456 (591h14m45.911078511s) [seg_pos={0 0}, upload=518325]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 30 06:43:46.040216396 (591h14m45.911299441s) [upload=518325]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 06:43:46.040225283 (591h14m45.911308328s) [upload=518325]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 06:43:46.331063484 (591h14m46.202146549s) [upload=518325]: Done uploading segments...
Dec 30 06:43:46.331089314 (591h14m46.202172349s) [upload=518325]: Upload finished. err: <nil>
Duration: 5.914187191s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
9 out of 60 is failed (threshold=6) [.....X...........X..............XX...X......X..X..X.....X...] duration:
3.574508899s trace:
https://jaeger.dev.storj.io/trace/7d0f3c444f9c906c
FAIL (2025-12-30 06:44:53 UTC)
orbiter warning: context deadline exceeded
Duration: 3.574537403s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [..................X.........................................] duration:
11.646484528s trace:
https://jaeger.dev.storj.io/trace/deb96436003dee
FAIL (2025-12-30 06:44:50 UTC)
orbiter warning: read tcp 188.166.225.170:52458->45.38.24.2:443: read: connection reset by peer
Duration: 11.646562756s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
4 out of 60 is failed (threshold=3) [.......X...X......X............X............................] duration:
33.412758949s trace:
https://jaeger.dev.storj.io/trace/45b769fc71e13448
FAIL (2025-12-30 06:44:44 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: context deadline exceeded
======LOGS======
.577697683 (590h53m11.448780728s) [noise=true , address=193.149.164.143:4704 , piece_id=4T2IJIA6WHI5LMIUP5BLJLCD3GRY3A6PXR6G5OWJWUSYZT5ELGHQ, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.578173528 (590h53m11.449256573s) [noise=true , address=64.49.13.27:4965 , piece_id=CO4O5F2IBZAAKBZBV3HWNNOXZOGLNNDPALWMQZ3DEM3MREASVFEA, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.578542992 (590h53m11.449626027s) [noise=true , address=77.90.31.15:4729 , piece_id=NVMXFEILHGT622IIPYYT4FFHC2WZLHMCZM4VT5WUPHRAPH3DX6TA, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.578846911 (590h53m11.449929946s) [noise=true , address=145.79.227.43:4177 , piece_id=YM37ZI3WNZ7SXGTFC224TOVVTHYNFIPRQB2TFLEZ735RNZE553FQ, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.579458044 (590h53m11.450541089s) [noise=true , address=45.91.116.11:4150 , piece_id=E66IDHTJDLYLYJIOWMOIKOOWLXCUNCMBIQ7SQBAD3VTJF6VX5JLQ, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.579891209 (590h53m11.450974245s) [noise=true , address=145.79.227.29:4945 , piece_id=SHL73IDEI54GRCWYNWHRUSJZZ4UUXREAX54DFJLTG5QPZCF5QKIQ, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.579915535 (590h53m11.45099858s) [noise=true , address=145.79.227.30:4772 , piece_id=PVMD5DWRRPVARTSZ6O7HN6MUVTZUYWLYSNQR5GDMNKUJZGXMYPFQ, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.580325466 (590h53m11.451408511s) [noise=true , address=77.90.31.12:4603 , piece_id=2I4L7F2WEGK5OSLODJSJ4672URBA7DUGXWZKTYB6ZNA2TQIGVXWA, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.580379599 (590h53m11.451462644s) [noise=true , address=77.90.31.14:4671 , piece_id=263TRISJ5ADFTX43ULSSGBAF2KMOXAPISJ56LLE53D74XOEMQ5EA, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.580692044 (590h53m11.451775089s) [noise=true , address=145.79.227.3:4175 , piece_id=UCV5WQNVWEAMEDBWDNXEVSQEGXPJE7KD6YSWC5A4WXYUKUCK4NLA, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.580857859 (590h53m11.451940894s) [noise=true , address=77.90.31.19:4860 , piece_id=U2MNFOAD7FHCDBE4FUYOROIP3P33ROJ7K75SF5Z747EFKFTF7RGA, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.581089210 (590h53m11.452172255s) [noise=true , address=145.79.227.44:4203 , piece_id=WHCKQ2FRLYAZE7Q6HYZNTS4EISVHZNAPDDKH34WN33ALK4MVK2EA, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.581351320 (590h53m11.452434355s) [noise=true , address=64.49.13.154:4928 , piece_id=SEVBZBMNGURP5TRUEBA4PHRSA6MNANVJUKINALJUWPQWBGKVYNTA, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.581457091 (590h53m11.452540126s) [noise=true , address=145.79.227.34:4909 , piece_id=3QSWTNR5LGUUAY3N5HJKSPL3TRQ6WO5F5OD4VTXKWHWUXUF7PVVQ, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.581863144 (590h53m11.452946179s) [noise=true , address=145.79.227.33:4881 , piece_id=EX66JF7U7A4ZSRWJMP4MCLQRPW2ZZFZFIUNV72XZXU7E5O3KWX3A, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.584010443 (590h53m11.455093478s) [noise=true , address=145.79.227.40:4070 , piece_id=YOAGDXNN4TTCK35LLQDSP24BOWLZXGNSQU7WMKEEVVMPXDVI74JA, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.585122791 (590h53m11.456205826s) [noise=true , address=45.91.116.14:4217 , piece_id=L4MAHTGKTNWBSN3YTB74E3DBJWVMA3DPU57MQK5FNK35DXRZ6AIA, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:12.983305402 (590h53m12.854388447s) [upload=517998]: Next returned nil segment
Dec 30 06:22:12.983321313 (590h53m12.854404348s) [upload=517998]: Waiting for error group managing segments...
Dec 30 06:22:13.379703053 (590h53m13.250786088s) [noise=true , address=64.49.13.27:4965 , piece_id=CO4O5F2IBZAAKBZBV3HWNNOXZOGLNNDPALWMQZ3DEM3MREASVFEA, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:13.418055622 (590h53m13.289138657s) [noise=true , address=145.79.227.3:4175 , piece_id=UCV5WQNVWEAMEDBWDNXEVSQEGXPJE7KD6YSWC5A4WXYUKUCK4NLA, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:13.509034160 (590h53m13.380117205s) [noise=true , address=64.49.13.154:4928 , piece_id=SEVBZBMNGURP5TRUEBA4PHRSA6MNANVJUKINALJUWPQWBGKVYNTA, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:13.642886154 (590h53m13.513969199s) [noise=true , address=77.90.31.14:4671 , piece_id=263TRISJ5ADFTX43ULSSGBAF2KMOXAPISJ56LLE53D74XOEMQ5EA, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:13.788145520 (590h53m13.659228555s) [noise=true , address=77.90.31.12:4603 , piece_id=2I4L7F2WEGK5OSLODJSJ4672URBA7DUGXWZKTYB6ZNA2TQIGVXWA, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:13.858878757 (590h53m13.729961792s) [noise=true , address=145.79.227.49:4369 , piece_id=XUEXGNJJIJ4DUBDNUV7CKF6J5JYHSLZAEPVGEDGBUBY42BIP4XEQ, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:13.879550569 (590h53m13.750633614s) [noise=true , address=145.79.227.34:4909 , piece_id=3QSWTNR5LGUUAY3N5HJKSPL3TRQ6WO5F5OD4VTXKWHWUXUF7PVVQ, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:13.940897723 (590h53m13.811980769s) [noise=true , address=77.90.31.19:4860 , piece_id=U2MNFOAD7FHCDBE4FUYOROIP3P33ROJ7K75SF5Z747EFKFTF7RGA, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:13.989791936 (590h53m13.860874981s) [noise=true , address=45.91.116.11:4150 , piece_id=E66IDHTJDLYLYJIOWMOIKOOWLXCUNCMBIQ7SQBAD3VTJF6VX5JLQ, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:14.289516718 (590h53m14.160599753s) [noise=true , address=145.79.227.44:4203 , piece_id=WHCKQ2FRLYAZE7Q6HYZNTS4EISVHZNAPDDKH34WN33ALK4MVK2EA, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:14.338651537 (590h53m14.209734582s) [noise=true , address=77.90.31.15:4729 , piece_id=NVMXFEILHGT622IIPYYT4FFHC2WZLHMCZM4VT5WUPHRAPH3DX6TA, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:14.344145168 (590h53m14.215228213s) [noise=true , address=145.79.227.30:4772 , piece_id=PVMD5DWRRPVARTSZ6O7HN6MUVTZUYWLYSNQR5GDMNKUJZGXMYPFQ, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:14.667232847 (590h53m14.538315892s) [noise=true , address=193.149.164.143:4704 , piece_id=4T2IJIA6WHI5LMIUP5BLJLCD3GRY3A6PXR6G5OWJWUSYZT5ELGHQ, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:14.671388990 (590h53m14.542472025s) [noise=true , address=145.79.227.29:4945 , piece_id=SHL73IDEI54GRCWYNWHRUSJZZ4UUXREAX54DFJLTG5QPZCF5QKIQ, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:14.671415190 (590h53m14.542498226s) [seg_pos={0 0}, upload=517998]: Segment reached optimal threshold of 14 pieces.
Dec 30 06:22:14.671803600 (590h53m14.542886655s) [noise=true , address=145.79.227.40:4070 , piece_id=YOAGDXNN4TTCK35LLQDSP24BOWLZXGNSQU7WMKEEVVMPXDVI74JA, seg_pos={0 0}, upload=517998]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12hU1LwL2gX7rVnrNgzaBCYB37ziqf7s7aUgtF2Uc3antReQ6me): protocol: expected piece hash; context canceled; EOF
Dec 30 06:22:14.671814470 (590h53m14.542897525s) [noise=true , address=145.79.227.40:4070 , piece_id=YOAGDXNN4TTCK35LLQDSP24BOWLZXGNSQU7WMKEEVVMPXDVI74JA, seg_pos={0 0}, upload=517998]: Long tail cancellation due to optimal threshold
Dec 30 06:22:14.672014932 (590h53m14.543097977s) [noise=true , address=145.79.227.43:4177 , piece_id=YM37ZI3WNZ7SXGTFC224TOVVTHYNFIPRQB2TFLEZ735RNZE553FQ, seg_pos={0 0}, upload=517998]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bhN3cR22QuAYihnnB3x3R97UZRWJ8sFPP8fjQcXZ7qhNZphZG): protocol: expected piece hash; context canceled; EOF
Dec 30 06:22:14.672022637 (590h53m14.543105672s) [noise=true , address=145.79.227.43:4177 , piece_id=YM37ZI3WNZ7SXGTFC224TOVVTHYNFIPRQB2TFLEZ735RNZE553FQ, seg_pos={0 0}, upload=517998]: Long tail cancellation due to optimal threshold
Dec 30 06:22:14.672088933 (590h53m14.543171988s) [noise=true , address=145.79.227.33:4881 , piece_id=EX66JF7U7A4ZSRWJMP4MCLQRPW2ZZFZFIUNV72XZXU7E5O3KWX3A, seg_pos={0 0}, upload=517998]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mMBjg1b4MqTZPE5NpNNMYDzmsHqWYotE5jBNmZLyA5FyYLZRp): protocol: expected piece hash; context canceled; EOF
Dec 30 06:22:14.672127677 (590h53m14.543210742s) [noise=true , address=145.79.227.33:4881 , piece_id=EX66JF7U7A4ZSRWJMP4MCLQRPW2ZZFZFIUNV72XZXU7E5O3KWX3A, seg_pos={0 0}, upload=517998]: Long tail cancellation due to optimal threshold
Dec 30 06:22:14.672312739 (590h53m14.543395774s) [noise=true , address=45.91.116.14:4217 , piece_id=L4MAHTGKTNWBSN3YTB74E3DBJWVMA3DPU57MQK5FNK35DXRZ6AIA, seg_pos={0 0}, upload=517998]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xSXWFiq27Dd2vBDD9vVnhDHVG59UmDKn3hVAhf3oyo5goVNaE): protocol: expected piece hash; context canceled; EOF
Dec 30 06:22:14.672319953 (590h53m14.543402988s) [noise=true , address=45.91.116.14:4217 , piece_id=L4MAHTGKTNWBSN3YTB74E3DBJWVMA3DPU57MQK5FNK35DXRZ6AIA, seg_pos={0 0}, upload=517998]: Long tail cancellation due to optimal threshold
Dec 30 06:22:14.672337847 (590h53m14.543420882s) [seg_pos={0 0}, upload=517998]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 30 06:22:14.672452947 (590h53m14.543535982s) [upload=517998]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 06:22:14.672458507 (590h53m14.543541542s) [upload=517998]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 06:22:14.981437069 (590h53m14.852520114s) [upload=517998]: Done uploading segments...
Dec 30 06:22:14.981456546 (590h53m14.852539581s) [upload=517998]: Upload finished. err: <nil>
Duration: 33.412783015s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
9 out of 60 is failed (threshold=6) [......X...........X..............XX...X......X..X..X.....X..] duration:
4.843980631s trace:
https://jaeger.dev.storj.io/trace/7d0f3c444f9c906c
FAIL (2025-12-30 06:44:30 UTC)
orbiter warning: context deadline exceeded
Duration: 4.844010887s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)antiblip:
1 out of 60 is failed (threshold=3) [........................................................X...] duration:
10.100985992s trace:
https://jaeger.dev.storj.io/trace/4c2f4efaff0e0ea0
FAIL (2025-12-30 06:44:22 UTC)
orbiter warning:
operation: DeleteObject
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
load=518292]: Uploading piece...
Dec 30 06:41:13.598100152 (591h12m13.469183187s) [noise=true , address=81.171.14.70:4041 , piece_id=NORNA6WFTGPNOHU7MPTNNB5ONTBS5DJGCN2NWIZAEPYSUSIOSGEQ, seg_pos={0 0}, upload=518292]: Uploading piece...
Dec 30 06:41:13.598587971 (591h12m13.469671016s) [noise=true , address=89.149.201.147:4007 , piece_id=HZLSYGSBP4TJH5V5XF3Q3AETJU625ERNDBD3VXW2HUZTTPAHLBMA, seg_pos={0 0}, upload=518292]: Uploading piece...
Dec 30 06:41:13.599093143 (591h12m13.470176178s) [noise=true , address=89.149.201.147:4008 , piece_id=XYFRNJVIFFZA5S36KFPGZPVCPLRL2OW5QBJD3UR37VPPA2RX3RJA, seg_pos={0 0}, upload=518292]: Uploading piece...
Dec 30 06:41:13.599699999 (591h12m13.470783034s) [noise=true , address=81.171.14.70:4039 , piece_id=DEJDXFDHOSAMIP2FN5LJBJNBB4QLJSE2AUDW6IMA6IC4VRY7VAGQ, seg_pos={0 0}, upload=518292]: Uploading piece...
Dec 30 06:41:13.600136169 (591h12m13.471219204s) [noise=true , address=185.167.204.205:4034 , piece_id=OWZHQB5ZDT2QZ2QAHZJVRWQ222MX6CWLKJUOIAAGSMAVLWGJAOBA, seg_pos={0 0}, upload=518292]: Uploading piece...
Dec 30 06:41:13.600543184 (591h12m13.471626229s) [noise=true , address=185.167.204.206:4058 , piece_id=ROOCGK64M5RHIWNAVCSZNYTZ6M2ORWB7MJ27XJVDTFVQIKJZHY4A, seg_pos={0 0}, upload=518292]: Uploading piece...
Dec 30 06:41:13.601048286 (591h12m13.472131332s) [noise=true , address=81.171.14.70:4046 , piece_id=B2MTUNKMUP5S7XW4SVQ5QI2ZO3VFB5QVAKBBY4BIZIUILM6O7VYQ, seg_pos={0 0}, upload=518292]: Uploading piece...
Dec 30 06:41:13.601698784 (591h12m13.472781819s) [noise=true , address=185.167.204.206:4064 , piece_id=H4PP7HPHXRX2LOZ4776AMWAJDN7OC435E2K4BXI7XOZYKTB4YUNA, seg_pos={0 0}, upload=518292]: Uploading piece...
Dec 30 06:41:13.602168259 (591h12m13.473251304s) [noise=true , address=185.167.204.206:4065 , piece_id=LD3J4656Q5QEUCMJ5OKEYATEVRIKSXO7XGARTN25BVM7FDIYOWKA, seg_pos={0 0}, upload=518292]: Uploading piece...
Dec 30 06:41:13.602782708 (591h12m13.473865753s) [noise=true , address=81.171.14.70:4027 , piece_id=6WEORWX5QOMFSJUEADIIBHHIPKNFP4NJPZAJADBUTPUSG7MZJMOQ, seg_pos={0 0}, upload=518292]: Uploading piece...
Dec 30 06:41:14.423395002 (591h12m14.294478047s) [upload=518292]: Next returned nil segment
Dec 30 06:41:14.423414068 (591h12m14.294497103s) [upload=518292]: Waiting for error group managing segments...
Dec 30 06:41:14.817650316 (591h12m14.688733351s) [noise=true , address=185.167.204.205:4029 , piece_id=KUOMMYJ2F4NIK6VN4C5M3Y3TMHYEZH7QFNJ6K3CLOAVI5R6VKTHQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:14.817870054 (591h12m14.688953099s) [noise=true , address=185.167.204.205:4019 , piece_id=H6AH654VGX6NHBUKODIWGL7I2RKE3DDMY4UM6VDONP2EP3NPDJBA, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:14.818623698 (591h12m14.689706744s) [noise=true , address=185.167.204.206:4065 , piece_id=LD3J4656Q5QEUCMJ5OKEYATEVRIKSXO7XGARTN25BVM7FDIYOWKA, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:14.818932436 (591h12m14.690015481s) [noise=true , address=185.167.204.206:4061 , piece_id=T2XS4IFPQSVS3T45SPQ5ZDDVT2VMTMXHKL2ZMX4OIOPRR67SCD2Q, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:14.819147046 (591h12m14.690230091s) [noise=true , address=185.167.204.206:4071 , piece_id=QI2TOY5KCDZVWCCLMAJXVUWD2HB7S7IOTEAV2EHAMUNHKOFKTNYA, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:14.819678578 (591h12m14.690761623s) [noise=true , address=185.167.204.206:4059 , piece_id=LMOGTWX2YIRR3EKNROK43O2B27S5L5U6GYB7HBVRBPQPH4WDDWZQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:14.819905028 (591h12m14.690988063s) [noise=true , address=185.167.204.206:4058 , piece_id=ROOCGK64M5RHIWNAVCSZNYTZ6M2ORWB7MJ27XJVDTFVQIKJZHY4A, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:14.820479312 (591h12m14.691562357s) [noise=true , address=185.167.204.205:4021 , piece_id=KP7J5XSY4SKSKZG7XXFZMNKMRP4472PXI7CW2UQBZICN4MAFO5RQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:14.823149296 (591h12m14.694232331s) [noise=true , address=185.167.204.205:4034 , piece_id=OWZHQB5ZDT2QZ2QAHZJVRWQ222MX6CWLKJUOIAAGSMAVLWGJAOBA, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:14.827906996 (591h12m14.698990041s) [noise=true , address=185.167.204.206:4055 , piece_id=VYEDR6XHZBZMTVJ5U5OF67WZOGVMCMQLXXBQUI7DHBCCVQ2RRERQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:14.828003769 (591h12m14.699086814s) [noise=true , address=185.167.204.205:4016 , piece_id=I6MUDLAXUUDJL2C4HFE22C2L56A7D6HSCNWXKWJ7OKO2N4RA4W7Q, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:14.828893203 (591h12m14.699976238s) [noise=true , address=185.167.204.206:4064 , piece_id=H4PP7HPHXRX2LOZ4776AMWAJDN7OC435E2K4BXI7XOZYKTB4YUNA, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:14.829477074 (591h12m14.700560109s) [noise=true , address=185.167.204.205:4017 , piece_id=CZP65KETMGOHFDMV7KENRL4V36UHBPWRDCBBPUM5NE5L2PV4AQCQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:14.829774972 (591h12m14.700858007s) [noise=true , address=185.167.204.206:4043 , piece_id=G7VZYID4TXGNPDN63NQULXOVSX42WMFFNPBX6KQJ5L2WSW4D2YGQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:14.831734493 (591h12m14.702817528s) [noise=true , address=185.167.204.206:4049 , piece_id=BQ7JEGSGPIFJUGZFBYRPDMBNPZ24BG6TEG4VUM4JUBTE4T2RF4SA, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:15.405688561 (591h12m15.276771606s) [noise=true , address=81.171.14.70:4027 , piece_id=6WEORWX5QOMFSJUEADIIBHHIPKNFP4NJPZAJADBUTPUSG7MZJMOQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:15.417110440 (591h12m15.288193485s) [noise=true , address=81.171.14.70:4031 , piece_id=U6Z4SNK32ZSWQ7NDR7ACXGRC4BADDOPR65ILBR64QLGVNDBADFFQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:15.425600446 (591h12m15.296683481s) [noise=true , address=81.171.14.70:4037 , piece_id=6A25LEIAQEJB3DNCTXQT6K7ZSKIGOW4QUGMNKLOG3CT2XXH25PSA, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:15.499056228 (591h12m15.370139263s) [noise=true , address=81.171.14.70:4038 , piece_id=3WRHW3VGXWHOEOV5Q6CLYSF7MUVWS7QHTNHLSTCKAUERPHAH3I3Q, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:15.514998925 (591h12m15.38608197s) [noise=true , address=81.171.14.70:4041 , piece_id=NORNA6WFTGPNOHU7MPTNNB5ONTBS5DJGCN2NWIZAEPYSUSIOSGEQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:15.528579544 (591h12m15.399662589s) [noise=true , address=81.171.14.70:4030 , piece_id=KITOJIT34CHS2QZBXCW2SJPZB4RHEHCMJ6IYQ4Z5KLL24YAWBIUQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:15.528991398 (591h12m15.400074434s) [noise=true , address=81.171.14.70:4047 , piece_id=ZWV5DP4YC3MSYDWEQH4ZK3VR4WZ4DUTI4VL3WVLQKFIEKIW44LOQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:15.534844544 (591h12m15.405927579s) [noise=true , address=81.171.14.70:4035 , piece_id=XCLL5SOAPJSSGF5YYQH6ZKGOHMLS6RI6H26FOW5XIJRY3TWYNEGA, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:15.915614910 (591h12m15.786697945s) [noise=true , address=81.171.14.70:4039 , piece_id=DEJDXFDHOSAMIP2FN5LJBJNBB4QLJSE2AUDW6IMA6IC4VRY7VAGQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:15.967820176 (591h12m15.838903211s) [noise=true , address=81.171.14.70:4046 , piece_id=B2MTUNKMUP5S7XW4SVQ5QI2ZO3VFB5QVAKBBY4BIZIUILM6O7VYQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:16.172116128 (591h12m16.043199163s) [noise=true , address=89.149.201.147:4007 , piece_id=HZLSYGSBP4TJH5V5XF3Q3AETJU625ERNDBD3VXW2HUZTTPAHLBMA, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:16.172137559 (591h12m16.043220604s) [seg_pos={0 0}, upload=518292]: Segment reached optimal threshold of 26 pieces.
Dec 30 06:41:16.172336949 (591h12m16.043419994s) [noise=true , address=89.149.201.147:4008 , piece_id=XYFRNJVIFFZA5S36KFPGZPVCPLRL2OW5QBJD3UR37VPPA2RX3RJA, seg_pos={0 0}, upload=518292]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QN8VP6o7qGw9Dd8EafU9xe7s7Cuf1FyqPMbnRaznrfPxfEUUn): protocol: expected piece hash; context canceled; EOF
Dec 30 06:41:16.172344213 (591h12m16.043427258s) [noise=true , address=89.149.201.147:4008 , piece_id=XYFRNJVIFFZA5S36KFPGZPVCPLRL2OW5QBJD3UR37VPPA2RX3RJA, seg_pos={0 0}, upload=518292]: Long tail cancellation due to optimal threshold
Dec 30 06:41:16.172554002 (591h12m16.043637037s) [noise=true , address=89.149.201.147:4020 , piece_id=N7G5DGEDYVKBTO5S6BYW3IYEOK7HAMK5JQIBMVPMPYRDHXODWEGA, seg_pos={0 0}, upload=518292]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12YeKHJH7CoouLFJYENJhzVLNkA32hJaqjLz3dXhr78eaSGtcic): protocol: expected piece hash; context canceled; EOF
Dec 30 06:41:16.172565233 (591h12m16.043648278s) [noise=true , address=89.149.201.147:4020 , piece_id=N7G5DGEDYVKBTO5S6BYW3IYEOK7HAMK5JQIBMVPMPYRDHXODWEGA, seg_pos={0 0}, upload=518292]: Long tail cancellation due to optimal threshold
Dec 30 06:41:16.172587506 (591h12m16.043670541s) [seg_pos={0 0}, upload=518292]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil>
Dec 30 06:41:16.172717945 (591h12m16.04380098s) [upload=518292]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 06:41:16.172724567 (591h12m16.043807602s) [upload=518292]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 06:41:16.404092237 (591h12m16.275175292s) [upload=518292]: Done uploading segments...
Dec 30 06:41:16.404119940 (591h12m16.275202975s) [upload=518292]: Upload finished. err: <nil>
Duration: 10.101017122s
core
uplink
eu-select-1
eu1-placement-15
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [...................X........................................] duration:
15.157566965s trace:
https://jaeger.dev.storj.io/trace/deb96436003dee
FAIL (2025-12-30 06:44:15 UTC)
orbiter warning: read tcp 188.166.225.170:52458->45.38.24.2:443: read: connection reset by peer
Duration: 15.157682206s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
14 out of 60 is failed (threshold=3) [.....X..X......X....X...X..X.X.X.....X.......X.X.....X....XX] duration:
30.003050487s trace:
https://jaeger.dev.storj.io/trace/78056e8d4b3848bb
FAIL (2025-12-30 06:44:12 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
Dec 30 06:43:43.438454167 (591h14m43.309537212s) [upload=518325]: Starting upload
Dec 30 06:43:43.438541793 (591h14m43.309624849s) [upload=518325]: Done starting upload
Dec 30 06:43:43.438670769 (591h14m43.309753824s) [upload=518325]: Uploading segments...
Dec 30 06:43:43.438849339 (591h14m43.309932394s) [upload=518325]: Got next segment. Inline: false
Dec 30 06:43:43.438883805 (591h14m43.30996685s) [upload=518325]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 30 06:43:43.438889526 (591h14m43.309972561s) [upload=518325]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 30 06:43:43.682039201 (591h14m43.553122246s) [seg_pos={0 0}, upload=518325]: Begin upload segment...
Dec 30 06:43:43.682106038 (591h14m43.553189083s) [seg_pos={0 0}, upload=518325]: Done begin upload segment.
Dec 30 06:43:43.682152857 (591h14m43.553235902s) [noise=true , address=145.79.227.11:4732 , piece_id=WOMYTSZBK7KZRKDYYZA3ORJ5M6RCYJUHU2O7GXYY3CSQVFYE3S4A, seg_pos={0 0}, upload=518325]: Uploading piece...
Dec 30 06:43:43.682258869 (591h14m43.553341914s) [noise=true , address=77.90.31.17:4770 , piece_id=EZYUKNUOTNWRXEROZAVMPDNQ2EAYUBO3KP7X2GWSQXU6ZRQ5EJQQ, seg_pos={0 0}, upload=518325]: Uploading piece...
Dec 30 06:43:43.683367860 (591h14m43.554450906s) [noise=true , address=145.79.227.14:4838 , piece_id=LNTWJQBCQVT7WXSCH5EIII7DHI4SVOYIVKK4F7IXIKM4KMED2YMQ, seg_pos={0 0}, upload=518325]: Uploading piece...
Dec 30 06:43:43.683377639 (591h14m43.554460684s) [noise=true , address=145.79.227.36:4518 , piece_id=KWGCY63F2DZC5MNJIY5T73RQHOGDUNIOHOMFBXJXWIQXFQA73PYA, seg_pos={0 0}, upload=518325]: Uploading piece...
Dec 30 06:43:43.683832937 (591h14m43.554915972s) [noise=true , address=45.91.116.11:4120 , piece_id=VTJS4ZRPH6KEIEAYEQ4ODRTTGTX6BA2FWMZRTNT75OF7SH2DZUDA, seg_pos={0 0}, upload=518325]: Uploading piece...
Dec 30 06:43:43.684783196 (591h14m43.555866241s) [noise=true , address=145.79.227.18:4007 , piece_id=OWWYNA55RHUTUGUQT4Y2ZBI3VQAQQVNJTFKG6ALMRXXDLW3DW4AQ, seg_pos={0 0}, upload=518325]: Uploading piece...
Dec 30 06:43:43.685038492 (591h14m43.556121527s) [noise=true , address=145.79.227.3:4151 , piece_id=EJ6XIYN4PQ5KGO6EYP4YMQHXEZKZ2NHASAJTNVQ3VF7S5PNHCYKA, seg_pos={0 0}, upload=518325]: Uploading piece...
Dec 30 06:43:43.686167853 (591h14m43.557250908s) [noise=true , address=45.91.116.9:4263 , piece_id=AS5TS3VJSLQMF7ZHLF7B2VBTDVIB23LHUU63EGI2ENDXGEYPDV5A, seg_pos={0 0}, upload=518325]: Uploading piece...
Dec 30 06:43:43.686258695 (591h14m43.55734174s) [noise=true , address=145.79.227.29:4943 , piece_id=TDSZ3VSCCNAU73ZL2CS5XPWOCA6TRTZZKY7VDS2WSK3E2PFFTNMA, seg_pos={0 0}, upload=518325]: Uploading piece...
Dec 30 06:43:43.687362998 (591h14m43.558446043s) [noise=true , address=45.91.116.10:4083 , piece_id=AFTCCUF3L7226XLTU4GEFEKYGYQAQPJKHIPKX5N7MT2UCU45L2BQ, seg_pos={0 0}, upload=518325]: Uploading piece...
Dec 30 06:43:43.687636299 (591h14m43.558719334s) [noise=true , address=145.79.227.50:4374 , piece_id=W3NSQQDQ72JDIVU3LX7BQSWTCTEM3UAZLH2GU2AM7I7DWIRHFKZA, seg_pos={0 0}, upload=518325]: Uploading piece...
Dec 30 06:43:43.688312987 (591h14m43.559396022s) [noise=true , address=145.79.227.26:4654 , piece_id=J7RRD6V3GTKDEZEPPOUNPBAMD3Q4QHIG7T6Z7FE72EJAJKSDTGBQ, seg_pos={0 0}, upload=518325]: Uploading piece...
Dec 30 06:43:43.688619340 (591h14m43.559702385s) [noise=true , address=77.90.31.15:4697 , piece_id=ZQJMG5N6Z5DYDTEMF5EL5YQGQVGKPEWMG6RZN6EXMEJYXJUQT3WA, seg_pos={0 0}, upload=518325]: Uploading piece...
Dec 30 06:43:43.688729590 (591h14m43.559812635s) [noise=true , address=145.79.227.6:4257 , piece_id=IMXIK3WJDAMAF7QXSYTWHGLDHUGQIBXQ3WDIZX45L4ODC35H7ICQ, seg_pos={0 0}, upload=518325]: Uploading piece...
Dec 30 06:43:43.689649502 (591h14m43.560732537s) [noise=true , address=77.90.31.10:4517 , piece_id=JXYAH2NNMGP5LSLNUGVUCG3X5NDPBW3MQ6ISE6HV4XPY7Y2CKSNA, seg_pos={0 0}, upload=518325]: Uploading piece...
Dec 30 06:43:43.690303357 (591h14m43.561386402s) [noise=true , address=145.79.227.51:4410 , piece_id=HDGHD2ORSRHHSIJG23HYPYM7GATOSAOD24MYETBMKCICOSTFCJMA, seg_pos={0 0}, upload=518325]: Uploading piece...
Dec 30 06:43:45.296790318 (591h14m45.167873363s) [upload=518325]: Next returned nil segment
Dec 30 06:43:45.296952557 (591h14m45.168035602s) [upload=518325]: Waiting for error group managing segments...
Dec 30 06:43:45.672927161 (591h14m45.544010236s) [noise=true , address=145.79.227.50:4374 , piece_id=W3NSQQDQ72JDIVU3LX7BQSWTCTEM3UAZLH2GU2AM7I7DWIRHFKZA, seg_pos={0 0}, upload=518325]: Done uploading piece. err: <nil>
Dec 30 06:43:45.679981644 (591h14m45.551064679s) [noise=true , address=145.79.227.14:4838 , piece_id=LNTWJQBCQVT7WXSCH5EIII7DHI4SVOYIVKK4F7IXIKM4KMED2YMQ, seg_pos={0 0}, upload=518325]: Done uploading piece. err: <nil>
Dec 30 06:43:45.681994216 (591h14m45.553077251s) [noise=true , address=145.79.227.11:4732 , piece_id=WOMYTSZBK7KZRKDYYZA3ORJ5M6RCYJUHU2O7GXYY3CSQVFYE3S4A, seg_pos={0 0}, upload=518325]: Done uploading piece. err: <nil>
Dec 30 06:43:45.682769572 (591h14m45.553852607s) [noise=true , address=145.79.227.36:4518 , piece_id=KWGCY63F2DZC5MNJIY5T73RQHOGDUNIOHOMFBXJXWIQXFQA73PYA, seg_pos={0 0}, upload=518325]: Done uploading piece. err: <nil>
Dec 30 06:43:45.685368261 (591h14m45.556451296s) [noise=true , address=145.79.227.51:4410 , piece_id=HDGHD2ORSRHHSIJG23HYPYM7GATOSAOD24MYETBMKCICOSTFCJMA, seg_pos={0 0}, upload=518325]: Done uploading piece. err: <nil>
Dec 30 06:43:45.696508394 (591h14m45.567591429s) [noise=true , address=145.79.227.26:4654 , piece_id=J7RRD6V3GTKDEZEPPOUNPBAMD3Q4QHIG7T6Z7FE72EJAJKSDTGBQ, seg_pos={0 0}, upload=518325]: Done uploading piece. err: <nil>
Dec 30 06:43:45.698096838 (591h14m45.569179883s) [noise=true , address=145.79.227.6:4257 , piece_id=IMXIK3WJDAMAF7QXSYTWHGLDHUGQIBXQ3WDIZX45L4ODC35H7ICQ, seg_pos={0 0}, upload=518325]: Done uploading piece. err: <nil>
Dec 30 06:43:45.699199428 (591h14m45.570282463s) [noise=true , address=145.79.227.18:4007 , piece_id=OWWYNA55RHUTUGUQT4Y2ZBI3VQAQQVNJTFKG6ALMRXXDLW3DW4AQ, seg_pos={0 0}, upload=518325]: Done uploading piece. err: <nil>
Dec 30 06:43:45.758676632 (591h14m45.629759677s) [noise=true , address=145.79.227.3:4151 , piece_id=EJ6XIYN4PQ5KGO6EYP4YMQHXEZKZ2NHASAJTNVQ3VF7S5PNHCYKA, seg_pos={0 0}, upload=518325]: Done uploading piece. err: <nil>
Dec 30 06:43:45.903122208 (591h14m45.774205253s) [noise=true , address=77.90.31.10:4517 , piece_id=JXYAH2NNMGP5LSLNUGVUCG3X5NDPBW3MQ6ISE6HV4XPY7Y2CKSNA, seg_pos={0 0}, upload=518325]: Done uploading piece. err: <nil>
Dec 30 06:43:45.976028733 (591h14m45.847111778s) [noise=true , address=145.79.227.29:4943 , piece_id=TDSZ3VSCCNAU73ZL2CS5XPWOCA6TRTZZKY7VDS2WSK3E2PFFTNMA, seg_pos={0 0}, upload=518325]: Done uploading piece. err: <nil>
Dec 30 06:43:45.982909666 (591h14m45.853992711s) [noise=true , address=77.90.31.17:4770 , piece_id=EZYUKNUOTNWRXEROZAVMPDNQ2EAYUBO3KP7X2GWSQXU6ZRQ5EJQQ, seg_pos={0 0}, upload=518325]: Done uploading piece. err: <nil>
Dec 30 06:43:46.039347542 (591h14m45.910430587s) [noise=true , address=45.91.116.10:4083 , piece_id=AFTCCUF3L7226XLTU4GEFEKYGYQAQPJKHIPKX5N7MT2UCU45L2BQ, seg_pos={0 0}, upload=518325]: Done uploading piece. err: <nil>
Dec 30 06:43:46.039372420 (591h14m45.910455475s) [seg_pos={0 0}, upload=518325]: Segment reached optimal threshold of 13 pieces.
Dec 30 06:43:46.039598750 (591h14m45.910681805s) [noise=true , address=77.90.31.15:4697 , piece_id=ZQJMG5N6Z5DYDTEMF5EL5YQGQVGKPEWMG6RZN6EXMEJYXJUQT3WA, seg_pos={0 0}, upload=518325]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5): protocol: expected piece hash; context canceled; EOF
Dec 30 06:43:46.039606464 (591h14m45.91068951s) [noise=true , address=77.90.31.15:4697 , piece_id=ZQJMG5N6Z5DYDTEMF5EL5YQGQVGKPEWMG6RZN6EXMEJYXJUQT3WA, seg_pos={0 0}, upload=518325]: Long tail cancellation due to optimal threshold
Dec 30 06:43:46.039875217 (591h14m45.910958282s) [noise=true , address=45.91.116.9:4263 , piece_id=AS5TS3VJSLQMF7ZHLF7B2VBTDVIB23LHUU63EGI2ENDXGEYPDV5A, seg_pos={0 0}, upload=518325]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF): protocol: expected piece hash; context canceled; EOF
Dec 30 06:43:46.039883282 (591h14m45.910966327s) [noise=true , address=45.91.116.9:4263 , piece_id=AS5TS3VJSLQMF7ZHLF7B2VBTDVIB23LHUU63EGI2ENDXGEYPDV5A, seg_pos={0 0}, upload=518325]: Long tail cancellation due to optimal threshold
Dec 30 06:43:46.039942615 (591h14m45.91102568s) [noise=true , address=45.91.116.11:4120 , piece_id=VTJS4ZRPH6KEIEAYEQ4ODRTTGTX6BA2FWMZRTNT75OF7SH2DZUDA, seg_pos={0 0}, upload=518325]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8): protocol: expected piece hash; context canceled; EOF
Dec 30 06:43:46.039962774 (591h14m45.911045829s) [noise=true , address=45.91.116.11:4120 , piece_id=VTJS4ZRPH6KEIEAYEQ4ODRTTGTX6BA2FWMZRTNT75OF7SH2DZUDA, seg_pos={0 0}, upload=518325]: Long tail cancellation due to optimal threshold
Dec 30 06:43:46.039995456 (591h14m45.911078511s) [seg_pos={0 0}, upload=518325]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 30 06:43:46.040216396 (591h14m45.911299441s) [upload=518325]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 06:43:46.040225283 (591h14m45.911308328s) [upload=518325]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 06:43:46.331063484 (591h14m46.202146549s) [upload=518325]: Done uploading segments...
Dec 30 06:43:46.331089314 (591h14m46.202172349s) [upload=518325]: Upload finished. err: <nil>
Duration: 30.003072208s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
9 out of 60 is failed (threshold=6) [.......X...........X..............XX...X......X..X..X.....X.] duration:
3.650327237s trace:
https://jaeger.dev.storj.io/trace/7d0f3c444f9c906c
FAIL (2025-12-30 06:44:12 UTC)
orbiter warning: context deadline exceeded
Duration: 3.650353809s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
4 out of 60 is failed (threshold=3) [........X...X......X............X...........................] duration:
51.831597174s trace:
https://jaeger.dev.storj.io/trace/45b769fc71e13448
FAIL (2025-12-30 06:44:10 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: context deadline exceeded
======LOGS======
.577697683 (590h53m11.448780728s) [noise=true , address=193.149.164.143:4704 , piece_id=4T2IJIA6WHI5LMIUP5BLJLCD3GRY3A6PXR6G5OWJWUSYZT5ELGHQ, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.578173528 (590h53m11.449256573s) [noise=true , address=64.49.13.27:4965 , piece_id=CO4O5F2IBZAAKBZBV3HWNNOXZOGLNNDPALWMQZ3DEM3MREASVFEA, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.578542992 (590h53m11.449626027s) [noise=true , address=77.90.31.15:4729 , piece_id=NVMXFEILHGT622IIPYYT4FFHC2WZLHMCZM4VT5WUPHRAPH3DX6TA, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.578846911 (590h53m11.449929946s) [noise=true , address=145.79.227.43:4177 , piece_id=YM37ZI3WNZ7SXGTFC224TOVVTHYNFIPRQB2TFLEZ735RNZE553FQ, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.579458044 (590h53m11.450541089s) [noise=true , address=45.91.116.11:4150 , piece_id=E66IDHTJDLYLYJIOWMOIKOOWLXCUNCMBIQ7SQBAD3VTJF6VX5JLQ, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.579891209 (590h53m11.450974245s) [noise=true , address=145.79.227.29:4945 , piece_id=SHL73IDEI54GRCWYNWHRUSJZZ4UUXREAX54DFJLTG5QPZCF5QKIQ, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.579915535 (590h53m11.45099858s) [noise=true , address=145.79.227.30:4772 , piece_id=PVMD5DWRRPVARTSZ6O7HN6MUVTZUYWLYSNQR5GDMNKUJZGXMYPFQ, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.580325466 (590h53m11.451408511s) [noise=true , address=77.90.31.12:4603 , piece_id=2I4L7F2WEGK5OSLODJSJ4672URBA7DUGXWZKTYB6ZNA2TQIGVXWA, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.580379599 (590h53m11.451462644s) [noise=true , address=77.90.31.14:4671 , piece_id=263TRISJ5ADFTX43ULSSGBAF2KMOXAPISJ56LLE53D74XOEMQ5EA, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.580692044 (590h53m11.451775089s) [noise=true , address=145.79.227.3:4175 , piece_id=UCV5WQNVWEAMEDBWDNXEVSQEGXPJE7KD6YSWC5A4WXYUKUCK4NLA, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.580857859 (590h53m11.451940894s) [noise=true , address=77.90.31.19:4860 , piece_id=U2MNFOAD7FHCDBE4FUYOROIP3P33ROJ7K75SF5Z747EFKFTF7RGA, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.581089210 (590h53m11.452172255s) [noise=true , address=145.79.227.44:4203 , piece_id=WHCKQ2FRLYAZE7Q6HYZNTS4EISVHZNAPDDKH34WN33ALK4MVK2EA, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.581351320 (590h53m11.452434355s) [noise=true , address=64.49.13.154:4928 , piece_id=SEVBZBMNGURP5TRUEBA4PHRSA6MNANVJUKINALJUWPQWBGKVYNTA, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.581457091 (590h53m11.452540126s) [noise=true , address=145.79.227.34:4909 , piece_id=3QSWTNR5LGUUAY3N5HJKSPL3TRQ6WO5F5OD4VTXKWHWUXUF7PVVQ, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.581863144 (590h53m11.452946179s) [noise=true , address=145.79.227.33:4881 , piece_id=EX66JF7U7A4ZSRWJMP4MCLQRPW2ZZFZFIUNV72XZXU7E5O3KWX3A, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.584010443 (590h53m11.455093478s) [noise=true , address=145.79.227.40:4070 , piece_id=YOAGDXNN4TTCK35LLQDSP24BOWLZXGNSQU7WMKEEVVMPXDVI74JA, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.585122791 (590h53m11.456205826s) [noise=true , address=45.91.116.14:4217 , piece_id=L4MAHTGKTNWBSN3YTB74E3DBJWVMA3DPU57MQK5FNK35DXRZ6AIA, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:12.983305402 (590h53m12.854388447s) [upload=517998]: Next returned nil segment
Dec 30 06:22:12.983321313 (590h53m12.854404348s) [upload=517998]: Waiting for error group managing segments...
Dec 30 06:22:13.379703053 (590h53m13.250786088s) [noise=true , address=64.49.13.27:4965 , piece_id=CO4O5F2IBZAAKBZBV3HWNNOXZOGLNNDPALWMQZ3DEM3MREASVFEA, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:13.418055622 (590h53m13.289138657s) [noise=true , address=145.79.227.3:4175 , piece_id=UCV5WQNVWEAMEDBWDNXEVSQEGXPJE7KD6YSWC5A4WXYUKUCK4NLA, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:13.509034160 (590h53m13.380117205s) [noise=true , address=64.49.13.154:4928 , piece_id=SEVBZBMNGURP5TRUEBA4PHRSA6MNANVJUKINALJUWPQWBGKVYNTA, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:13.642886154 (590h53m13.513969199s) [noise=true , address=77.90.31.14:4671 , piece_id=263TRISJ5ADFTX43ULSSGBAF2KMOXAPISJ56LLE53D74XOEMQ5EA, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:13.788145520 (590h53m13.659228555s) [noise=true , address=77.90.31.12:4603 , piece_id=2I4L7F2WEGK5OSLODJSJ4672URBA7DUGXWZKTYB6ZNA2TQIGVXWA, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:13.858878757 (590h53m13.729961792s) [noise=true , address=145.79.227.49:4369 , piece_id=XUEXGNJJIJ4DUBDNUV7CKF6J5JYHSLZAEPVGEDGBUBY42BIP4XEQ, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:13.879550569 (590h53m13.750633614s) [noise=true , address=145.79.227.34:4909 , piece_id=3QSWTNR5LGUUAY3N5HJKSPL3TRQ6WO5F5OD4VTXKWHWUXUF7PVVQ, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:13.940897723 (590h53m13.811980769s) [noise=true , address=77.90.31.19:4860 , piece_id=U2MNFOAD7FHCDBE4FUYOROIP3P33ROJ7K75SF5Z747EFKFTF7RGA, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:13.989791936 (590h53m13.860874981s) [noise=true , address=45.91.116.11:4150 , piece_id=E66IDHTJDLYLYJIOWMOIKOOWLXCUNCMBIQ7SQBAD3VTJF6VX5JLQ, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:14.289516718 (590h53m14.160599753s) [noise=true , address=145.79.227.44:4203 , piece_id=WHCKQ2FRLYAZE7Q6HYZNTS4EISVHZNAPDDKH34WN33ALK4MVK2EA, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:14.338651537 (590h53m14.209734582s) [noise=true , address=77.90.31.15:4729 , piece_id=NVMXFEILHGT622IIPYYT4FFHC2WZLHMCZM4VT5WUPHRAPH3DX6TA, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:14.344145168 (590h53m14.215228213s) [noise=true , address=145.79.227.30:4772 , piece_id=PVMD5DWRRPVARTSZ6O7HN6MUVTZUYWLYSNQR5GDMNKUJZGXMYPFQ, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:14.667232847 (590h53m14.538315892s) [noise=true , address=193.149.164.143:4704 , piece_id=4T2IJIA6WHI5LMIUP5BLJLCD3GRY3A6PXR6G5OWJWUSYZT5ELGHQ, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:14.671388990 (590h53m14.542472025s) [noise=true , address=145.79.227.29:4945 , piece_id=SHL73IDEI54GRCWYNWHRUSJZZ4UUXREAX54DFJLTG5QPZCF5QKIQ, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:14.671415190 (590h53m14.542498226s) [seg_pos={0 0}, upload=517998]: Segment reached optimal threshold of 14 pieces.
Dec 30 06:22:14.671803600 (590h53m14.542886655s) [noise=true , address=145.79.227.40:4070 , piece_id=YOAGDXNN4TTCK35LLQDSP24BOWLZXGNSQU7WMKEEVVMPXDVI74JA, seg_pos={0 0}, upload=517998]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12hU1LwL2gX7rVnrNgzaBCYB37ziqf7s7aUgtF2Uc3antReQ6me): protocol: expected piece hash; context canceled; EOF
Dec 30 06:22:14.671814470 (590h53m14.542897525s) [noise=true , address=145.79.227.40:4070 , piece_id=YOAGDXNN4TTCK35LLQDSP24BOWLZXGNSQU7WMKEEVVMPXDVI74JA, seg_pos={0 0}, upload=517998]: Long tail cancellation due to optimal threshold
Dec 30 06:22:14.672014932 (590h53m14.543097977s) [noise=true , address=145.79.227.43:4177 , piece_id=YM37ZI3WNZ7SXGTFC224TOVVTHYNFIPRQB2TFLEZ735RNZE553FQ, seg_pos={0 0}, upload=517998]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bhN3cR22QuAYihnnB3x3R97UZRWJ8sFPP8fjQcXZ7qhNZphZG): protocol: expected piece hash; context canceled; EOF
Dec 30 06:22:14.672022637 (590h53m14.543105672s) [noise=true , address=145.79.227.43:4177 , piece_id=YM37ZI3WNZ7SXGTFC224TOVVTHYNFIPRQB2TFLEZ735RNZE553FQ, seg_pos={0 0}, upload=517998]: Long tail cancellation due to optimal threshold
Dec 30 06:22:14.672088933 (590h53m14.543171988s) [noise=true , address=145.79.227.33:4881 , piece_id=EX66JF7U7A4ZSRWJMP4MCLQRPW2ZZFZFIUNV72XZXU7E5O3KWX3A, seg_pos={0 0}, upload=517998]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mMBjg1b4MqTZPE5NpNNMYDzmsHqWYotE5jBNmZLyA5FyYLZRp): protocol: expected piece hash; context canceled; EOF
Dec 30 06:22:14.672127677 (590h53m14.543210742s) [noise=true , address=145.79.227.33:4881 , piece_id=EX66JF7U7A4ZSRWJMP4MCLQRPW2ZZFZFIUNV72XZXU7E5O3KWX3A, seg_pos={0 0}, upload=517998]: Long tail cancellation due to optimal threshold
Dec 30 06:22:14.672312739 (590h53m14.543395774s) [noise=true , address=45.91.116.14:4217 , piece_id=L4MAHTGKTNWBSN3YTB74E3DBJWVMA3DPU57MQK5FNK35DXRZ6AIA, seg_pos={0 0}, upload=517998]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xSXWFiq27Dd2vBDD9vVnhDHVG59UmDKn3hVAhf3oyo5goVNaE): protocol: expected piece hash; context canceled; EOF
Dec 30 06:22:14.672319953 (590h53m14.543402988s) [noise=true , address=45.91.116.14:4217 , piece_id=L4MAHTGKTNWBSN3YTB74E3DBJWVMA3DPU57MQK5FNK35DXRZ6AIA, seg_pos={0 0}, upload=517998]: Long tail cancellation due to optimal threshold
Dec 30 06:22:14.672337847 (590h53m14.543420882s) [seg_pos={0 0}, upload=517998]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 30 06:22:14.672452947 (590h53m14.543535982s) [upload=517998]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 06:22:14.672458507 (590h53m14.543541542s) [upload=517998]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 06:22:14.981437069 (590h53m14.852520114s) [upload=517998]: Done uploading segments...
Dec 30 06:22:14.981456546 (590h53m14.852539581s) [upload=517998]: Upload finished. err: <nil>
Duration: 51.831625269s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [....................X.......................................] duration:
6.267669655s trace:
https://jaeger.dev.storj.io/trace/deb96436003dee
FAIL (2025-12-30 06:43:48 UTC)
orbiter warning: read tcp 188.166.225.170:52458->45.38.24.2:443: read: connection reset by peer
Duration: 6.267732995s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
9 out of 60 is failed (threshold=6) [........X...........X..............XX...X......X..X..X.....X] duration:
5.001646253s trace:
https://jaeger.dev.storj.io/trace/7d0f3c444f9c906c
FAIL (2025-12-30 06:43:44 UTC)
orbiter warning: context deadline exceeded
Duration: 5.001668835s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)antiblip:
1 out of 60 is failed (threshold=3) [.........................................................X..] duration:
7.176797549s trace:
https://jaeger.dev.storj.io/trace/4c2f4efaff0e0ea0
FAIL (2025-12-30 06:43:31 UTC)
orbiter warning:
operation: DeleteObject
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
load=518292]: Uploading piece...
Dec 30 06:41:13.598100152 (591h12m13.469183187s) [noise=true , address=81.171.14.70:4041 , piece_id=NORNA6WFTGPNOHU7MPTNNB5ONTBS5DJGCN2NWIZAEPYSUSIOSGEQ, seg_pos={0 0}, upload=518292]: Uploading piece...
Dec 30 06:41:13.598587971 (591h12m13.469671016s) [noise=true , address=89.149.201.147:4007 , piece_id=HZLSYGSBP4TJH5V5XF3Q3AETJU625ERNDBD3VXW2HUZTTPAHLBMA, seg_pos={0 0}, upload=518292]: Uploading piece...
Dec 30 06:41:13.599093143 (591h12m13.470176178s) [noise=true , address=89.149.201.147:4008 , piece_id=XYFRNJVIFFZA5S36KFPGZPVCPLRL2OW5QBJD3UR37VPPA2RX3RJA, seg_pos={0 0}, upload=518292]: Uploading piece...
Dec 30 06:41:13.599699999 (591h12m13.470783034s) [noise=true , address=81.171.14.70:4039 , piece_id=DEJDXFDHOSAMIP2FN5LJBJNBB4QLJSE2AUDW6IMA6IC4VRY7VAGQ, seg_pos={0 0}, upload=518292]: Uploading piece...
Dec 30 06:41:13.600136169 (591h12m13.471219204s) [noise=true , address=185.167.204.205:4034 , piece_id=OWZHQB5ZDT2QZ2QAHZJVRWQ222MX6CWLKJUOIAAGSMAVLWGJAOBA, seg_pos={0 0}, upload=518292]: Uploading piece...
Dec 30 06:41:13.600543184 (591h12m13.471626229s) [noise=true , address=185.167.204.206:4058 , piece_id=ROOCGK64M5RHIWNAVCSZNYTZ6M2ORWB7MJ27XJVDTFVQIKJZHY4A, seg_pos={0 0}, upload=518292]: Uploading piece...
Dec 30 06:41:13.601048286 (591h12m13.472131332s) [noise=true , address=81.171.14.70:4046 , piece_id=B2MTUNKMUP5S7XW4SVQ5QI2ZO3VFB5QVAKBBY4BIZIUILM6O7VYQ, seg_pos={0 0}, upload=518292]: Uploading piece...
Dec 30 06:41:13.601698784 (591h12m13.472781819s) [noise=true , address=185.167.204.206:4064 , piece_id=H4PP7HPHXRX2LOZ4776AMWAJDN7OC435E2K4BXI7XOZYKTB4YUNA, seg_pos={0 0}, upload=518292]: Uploading piece...
Dec 30 06:41:13.602168259 (591h12m13.473251304s) [noise=true , address=185.167.204.206:4065 , piece_id=LD3J4656Q5QEUCMJ5OKEYATEVRIKSXO7XGARTN25BVM7FDIYOWKA, seg_pos={0 0}, upload=518292]: Uploading piece...
Dec 30 06:41:13.602782708 (591h12m13.473865753s) [noise=true , address=81.171.14.70:4027 , piece_id=6WEORWX5QOMFSJUEADIIBHHIPKNFP4NJPZAJADBUTPUSG7MZJMOQ, seg_pos={0 0}, upload=518292]: Uploading piece...
Dec 30 06:41:14.423395002 (591h12m14.294478047s) [upload=518292]: Next returned nil segment
Dec 30 06:41:14.423414068 (591h12m14.294497103s) [upload=518292]: Waiting for error group managing segments...
Dec 30 06:41:14.817650316 (591h12m14.688733351s) [noise=true , address=185.167.204.205:4029 , piece_id=KUOMMYJ2F4NIK6VN4C5M3Y3TMHYEZH7QFNJ6K3CLOAVI5R6VKTHQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:14.817870054 (591h12m14.688953099s) [noise=true , address=185.167.204.205:4019 , piece_id=H6AH654VGX6NHBUKODIWGL7I2RKE3DDMY4UM6VDONP2EP3NPDJBA, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:14.818623698 (591h12m14.689706744s) [noise=true , address=185.167.204.206:4065 , piece_id=LD3J4656Q5QEUCMJ5OKEYATEVRIKSXO7XGARTN25BVM7FDIYOWKA, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:14.818932436 (591h12m14.690015481s) [noise=true , address=185.167.204.206:4061 , piece_id=T2XS4IFPQSVS3T45SPQ5ZDDVT2VMTMXHKL2ZMX4OIOPRR67SCD2Q, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:14.819147046 (591h12m14.690230091s) [noise=true , address=185.167.204.206:4071 , piece_id=QI2TOY5KCDZVWCCLMAJXVUWD2HB7S7IOTEAV2EHAMUNHKOFKTNYA, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:14.819678578 (591h12m14.690761623s) [noise=true , address=185.167.204.206:4059 , piece_id=LMOGTWX2YIRR3EKNROK43O2B27S5L5U6GYB7HBVRBPQPH4WDDWZQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:14.819905028 (591h12m14.690988063s) [noise=true , address=185.167.204.206:4058 , piece_id=ROOCGK64M5RHIWNAVCSZNYTZ6M2ORWB7MJ27XJVDTFVQIKJZHY4A, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:14.820479312 (591h12m14.691562357s) [noise=true , address=185.167.204.205:4021 , piece_id=KP7J5XSY4SKSKZG7XXFZMNKMRP4472PXI7CW2UQBZICN4MAFO5RQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:14.823149296 (591h12m14.694232331s) [noise=true , address=185.167.204.205:4034 , piece_id=OWZHQB5ZDT2QZ2QAHZJVRWQ222MX6CWLKJUOIAAGSMAVLWGJAOBA, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:14.827906996 (591h12m14.698990041s) [noise=true , address=185.167.204.206:4055 , piece_id=VYEDR6XHZBZMTVJ5U5OF67WZOGVMCMQLXXBQUI7DHBCCVQ2RRERQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:14.828003769 (591h12m14.699086814s) [noise=true , address=185.167.204.205:4016 , piece_id=I6MUDLAXUUDJL2C4HFE22C2L56A7D6HSCNWXKWJ7OKO2N4RA4W7Q, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:14.828893203 (591h12m14.699976238s) [noise=true , address=185.167.204.206:4064 , piece_id=H4PP7HPHXRX2LOZ4776AMWAJDN7OC435E2K4BXI7XOZYKTB4YUNA, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:14.829477074 (591h12m14.700560109s) [noise=true , address=185.167.204.205:4017 , piece_id=CZP65KETMGOHFDMV7KENRL4V36UHBPWRDCBBPUM5NE5L2PV4AQCQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:14.829774972 (591h12m14.700858007s) [noise=true , address=185.167.204.206:4043 , piece_id=G7VZYID4TXGNPDN63NQULXOVSX42WMFFNPBX6KQJ5L2WSW4D2YGQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:14.831734493 (591h12m14.702817528s) [noise=true , address=185.167.204.206:4049 , piece_id=BQ7JEGSGPIFJUGZFBYRPDMBNPZ24BG6TEG4VUM4JUBTE4T2RF4SA, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:15.405688561 (591h12m15.276771606s) [noise=true , address=81.171.14.70:4027 , piece_id=6WEORWX5QOMFSJUEADIIBHHIPKNFP4NJPZAJADBUTPUSG7MZJMOQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:15.417110440 (591h12m15.288193485s) [noise=true , address=81.171.14.70:4031 , piece_id=U6Z4SNK32ZSWQ7NDR7ACXGRC4BADDOPR65ILBR64QLGVNDBADFFQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:15.425600446 (591h12m15.296683481s) [noise=true , address=81.171.14.70:4037 , piece_id=6A25LEIAQEJB3DNCTXQT6K7ZSKIGOW4QUGMNKLOG3CT2XXH25PSA, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:15.499056228 (591h12m15.370139263s) [noise=true , address=81.171.14.70:4038 , piece_id=3WRHW3VGXWHOEOV5Q6CLYSF7MUVWS7QHTNHLSTCKAUERPHAH3I3Q, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:15.514998925 (591h12m15.38608197s) [noise=true , address=81.171.14.70:4041 , piece_id=NORNA6WFTGPNOHU7MPTNNB5ONTBS5DJGCN2NWIZAEPYSUSIOSGEQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:15.528579544 (591h12m15.399662589s) [noise=true , address=81.171.14.70:4030 , piece_id=KITOJIT34CHS2QZBXCW2SJPZB4RHEHCMJ6IYQ4Z5KLL24YAWBIUQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:15.528991398 (591h12m15.400074434s) [noise=true , address=81.171.14.70:4047 , piece_id=ZWV5DP4YC3MSYDWEQH4ZK3VR4WZ4DUTI4VL3WVLQKFIEKIW44LOQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:15.534844544 (591h12m15.405927579s) [noise=true , address=81.171.14.70:4035 , piece_id=XCLL5SOAPJSSGF5YYQH6ZKGOHMLS6RI6H26FOW5XIJRY3TWYNEGA, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:15.915614910 (591h12m15.786697945s) [noise=true , address=81.171.14.70:4039 , piece_id=DEJDXFDHOSAMIP2FN5LJBJNBB4QLJSE2AUDW6IMA6IC4VRY7VAGQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:15.967820176 (591h12m15.838903211s) [noise=true , address=81.171.14.70:4046 , piece_id=B2MTUNKMUP5S7XW4SVQ5QI2ZO3VFB5QVAKBBY4BIZIUILM6O7VYQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:16.172116128 (591h12m16.043199163s) [noise=true , address=89.149.201.147:4007 , piece_id=HZLSYGSBP4TJH5V5XF3Q3AETJU625ERNDBD3VXW2HUZTTPAHLBMA, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:16.172137559 (591h12m16.043220604s) [seg_pos={0 0}, upload=518292]: Segment reached optimal threshold of 26 pieces.
Dec 30 06:41:16.172336949 (591h12m16.043419994s) [noise=true , address=89.149.201.147:4008 , piece_id=XYFRNJVIFFZA5S36KFPGZPVCPLRL2OW5QBJD3UR37VPPA2RX3RJA, seg_pos={0 0}, upload=518292]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QN8VP6o7qGw9Dd8EafU9xe7s7Cuf1FyqPMbnRaznrfPxfEUUn): protocol: expected piece hash; context canceled; EOF
Dec 30 06:41:16.172344213 (591h12m16.043427258s) [noise=true , address=89.149.201.147:4008 , piece_id=XYFRNJVIFFZA5S36KFPGZPVCPLRL2OW5QBJD3UR37VPPA2RX3RJA, seg_pos={0 0}, upload=518292]: Long tail cancellation due to optimal threshold
Dec 30 06:41:16.172554002 (591h12m16.043637037s) [noise=true , address=89.149.201.147:4020 , piece_id=N7G5DGEDYVKBTO5S6BYW3IYEOK7HAMK5JQIBMVPMPYRDHXODWEGA, seg_pos={0 0}, upload=518292]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12YeKHJH7CoouLFJYENJhzVLNkA32hJaqjLz3dXhr78eaSGtcic): protocol: expected piece hash; context canceled; EOF
Dec 30 06:41:16.172565233 (591h12m16.043648278s) [noise=true , address=89.149.201.147:4020 , piece_id=N7G5DGEDYVKBTO5S6BYW3IYEOK7HAMK5JQIBMVPMPYRDHXODWEGA, seg_pos={0 0}, upload=518292]: Long tail cancellation due to optimal threshold
Dec 30 06:41:16.172587506 (591h12m16.043670541s) [seg_pos={0 0}, upload=518292]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil>
Dec 30 06:41:16.172717945 (591h12m16.04380098s) [upload=518292]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 06:41:16.172724567 (591h12m16.043807602s) [upload=518292]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 06:41:16.404092237 (591h12m16.275175292s) [upload=518292]: Done uploading segments...
Dec 30 06:41:16.404119940 (591h12m16.275202975s) [upload=518292]: Upload finished. err: <nil>
Duration: 7.176819681s
core
uplink
eu-select-1
eu1-placement-15
|